Repository navigation
Feed problem, some videos do not show up #1130
Description
Official Instance
- The bug is reproducable on the official hosted instance or is API related.
Describe the bug
I self host piped.esmailelbob.xyz and I noticed videos from channels I subbed to in invidious show up and takes so long to show up in piped
To Reproduce
sub to some channels
take channels file
add it to both invidious and piped
Expected behavior
videos show up in my feed when some channel upload a video
Logs/Errors
hmm, not sure what to add here
Browser, and OS with Version.
No response
Additional context
No response
Activity
FireMasterK commentedon Jun 27, 2022
Do you have a list of channel IDs that are affected by this bug?
I can check their pubsub subscription status then at https://pubsubhubbub.appspot.com/
EsmailELBoBDev2 commentedon Jun 28, 2022
I can give you the sub file...
subscriptions.json.txt (had ti change it to .txt for github)
but I noticed something weird, I had problem with my instance I think it was problem with the database, while piped worked okay the login page did not work (pressing on login button did nothing) till I restarted the docker image so I assume it's problem in my own instance, somehow fetchers do not run right?
FireMasterK commentedon Jul 2, 2022
This should be fixed with TeamPiped/Piped-Backend@3a00940, let me know if it is not.
azR-103 commentedon Jul 2, 2022
Still have same issue, not all channel feed are being loaded, although i dont have this issue on this custom instance.
https://pipedus.palash.dev
FireMasterK commentedon Jul 2, 2022
Only new videos uploaded from 6 hours back would be updated in the feed, this is due to the nature of how PubSub works.
Let me know if this happens on new uploads from now.
EsmailELBoBDev2 commentedon Jul 3, 2022
@FireMasterK out of context issue, I noticed piped instance and login (I guess db) sometimes disconnect and I can't login without restart the whole instance so do I need to restart it every hour like invidious?
about feed because what happened, my sync stopped 3 days ago and now I'm waiting it to pull new videos and tell you it worked or no
EsmailELBoBDev2 commentedon Jul 3, 2022
I have to visit each channel to get new videos in my feed, weird behavior
Is there a command or way to force it to fetch updates from yt?
EsmailELBoBDev2 commentedon Jul 3, 2022
still do not work, as you see in these pics, in freetube it detected it but in piped I had to visit the channel to make it appear in my feed
FireMasterK commentedon Jul 19, 2022
Is there any logs that you can provide? run
docker-compose logs piped-backendThe pubsub debug shows that there has been no attempts to subscribe to feed updated for over a month (!!!)
Could you also provide you instance's
config.propertiesfile? Just ensure to redact any database informationSee: https://pubsubhubbub.appspot.com/subscription-details?hub.callback=https%3A%2F%2Fpipedapi.esmailelbob.xyz%2Fwebhooks%2Fpubsub&hub.topic=https%3A%2F%2Fwww.youtube.com%2Fxml%2Ffeeds%2Fvideos.xml%3Fchannel_id%3DUCadoKaQadkmNZg-MymcKWqg&hub.secret=
EsmailELBoBDev2 commentedon Jul 20, 2022
Im away from my pc so excuse my werid english
Umm weidly enoguh, i get
ERROR: No such service: piped-backendso hmmmAnd my config file
FireMasterK commentedon Jul 21, 2022
Try copying the name of the container from
docker psand replace it in thedocker logscommand.EsmailELBoBDev2 commentedon Jul 21, 2022
Hmm, i have piped backend running but i cant access its logs, it throws same error
FireMasterK commentedon Jul 21, 2022
Nvm, you have to run
docker logs piped-backend, not docker-compose, my bad!EsmailELBoBDev2 commentedon Jul 21, 2022
Ohhhhh 😂
Amyways i cant provide logs as im on phone and cant copy logs and when i send them to commandline pastebin it sends different output
68 remaining items
tron1point0 commentedon Mar 20, 2024
Then you can drop the
-s("silent") flag from thecurlcommand so it shows a progress indicator, like so:IngwiePhoenix commentedon Jan 7, 2025
I ran the script above (albeit modified, I use k3s) and the feed still is not updated.
Any other way I can solve this?
This number also shows up in the logs:
0xF4CED commentedon Jan 8, 2025
Here is a much improved script to do this.
It does not hammer the API with new HTTP handshakes for every channel.
It requests all channels in one HTTP session, and the server can decide how fast it will respond/process them.
Also, it does not request the content of each endpoint/channel but only asks for status information, further reducing load.
I have noticed that the piped API now takes exactly 1 second for every channel/request, which has not been the case previously.Is this a new rate limiting feature of piped, or is it youtubes rate limiting @FireMasterK?
Edit: my bad, wasn't piped related.
PhrozenByte commentedon Jan 9, 2025
@TeamPiped Since there seems to be a stable, scriptable workaround and since a proper fix is pending for 2,5 years now, could this workaround be included in the official Docker image, so that all Piped instances benefit from it?
The issue wasn't that serious 2,5 years ago because it only affected few videos, but since YouTube's war against adblockers and 3rd-party apps it became a permanent condition that renders Piped's subscription feature basically useless. This also affects popular dependant projects like LibreTube. Including this workaround into the official images would be very much appreciated.
tron1point0 commentedon Jan 9, 2025
Nice! I golfed that a little differently to pass the URLs to curl via
--configon STDIN because the total length of the command line is limited:The
sedcould also be removed to makepsqldo all the string munging, but I find this pipeline clearer.0xF4CED commentedon Jan 11, 2025
PhrozenByte commentedon Jan 11, 2025
I'd say every workaround is a hack, that's what distinguishes it from a proper fix 😄
The problem is that this issue persists for 2,5 years now and it seems like that nobody is either able (this includes myself), or willing to invest the time necessary to develop a proper fix, or it really can't be fixed in a meaningful way. I don't know and honestly, I don't think the reasons matter in the end. Since subscriptions is one of Piped's key features, I'd consider its breakage (and let's be honest, it's completely broken) major enough to justify a workaround until a proper fix is found and merged. That's just my IMHO though.
Since it's a workaround, downsides - even major ones - are expected. Workarounds are temporary solutions after all. The question is whether the workaround stays feasible.
Working with a rate limit doesn't really prevent us from applying this workaround, but only how many channels we can process in a given time. So for example, if users of a given instance have subscribed to 21,600 unique channels, feeds can only be updated four times a day max. I can only speak for myself here, but I'd definitely prefer having a feed with a 6 hour delay instead of having no working feed at all.
I don't run a public Piped instance, so I can't really assess about how many unique channels we're talking for larger instances. For example, kavin.rocks has 421,150 registered users. It's unlikely that subscriptions don't overlap, but I guess we still end up with way too many unique channels on kavin.rocks. So, this workaround likely isn't feasible for kavin.rocks, at least not until rate limit evasion measures are taken. The two next largest instances (adminforge.de and drgns.space) have 23,367 and 13,979 registered users respectively. I can only speculate, but I would be surprised (considering what I know now, there definitely could be obstacles I just don't know, please tell me then) if we can't at least get daily updates (= 86,400 unique channels max) with this workaround on these instances. Smaller instances should be fine and allow for more than one feed update per day.
So, let's work on improving the workaround to make it more feasible for instances of any size and ready to be included in the official Docker images.
First we need a config option to enable/disable this workaround. This could be as simple as adding a Docker env variable. Second we can't leave calling this script up to a cronjob, because we don't know whether the number of unique channels exceeds the number of channels we can process in the cronjob's frequency. On the other hand we don't want to update channels too often. So, let's modify the script so that it runs repeatedly indefinitely and pauses if it has processed all channels before the next run is due. The frequency should be configurable, too (let's start with hourly updates).
Here's a proof-of-concept with these two additions:
It still remains a rather simple workaround. That's by design, if it gets too complex it would be better to either develop a proper fix or at least include the workaround directly into Piped instead of calling Piped's API with
curl. However, we're not at this point yet.So, let's discuss a few more possible additions. You mentioned that it doesn't make much sense to update channels that don't publish videos that frequently. I absolutely agree. A simple solution could be to actually add two workaround scripts: One frequent version that excludes channels without recent videos (that's as simple as using a different SQL query), and another less frequently running version that only updates the channels excluded by the first one. This two-tier system could also include other criteria, for example channels that don't have many subscribers could fall into the low-frequency tier, or could be excluded completely (e.g. if there's just one subscriber). The criteria could be configurable, so that admins can choose what fits their instance best. WDYT?
alvv-z commentedon Feb 18, 2025
It might be possible to federate the feeds between instances to avoid excess requests if another instance has processed a channel fairly recently. Would be a much larger workaround than what @PhrozenByte suggested, however.
firejoust commentedon Apr 12, 2025
Unfortunately I've had no luck with the workaround on my instance. However, I made a fork based on it and it seems to do the trick for me:
https://github.com/firejoust/Piped-Backend
I would only recommend it for people using private instances. The instructions are provided in the README document for anyone interested.
PhrozenByte commentedon Apr 12, 2025
@firejoust Thanks for your work! 🚀 I didn't find any open PR with your changes. How about creating one and getting some feedback about your changes? Your approach might be generally useful or at least start a discussion about how a generally useful solution might look like 👍
Weidav commentedon Jun 23, 2025
Works like a charm!
Would you mind opening a PR as @PhrozenByte suggested?
peetezaah commentedon Jul 21, 2025
@firejoust , @PhrozenByte and @Weidav have you found any other issues?
Because I used the workaround and now seem to have another issue: seeking breaks the video entirely, if I jump from the beginning of a video into, for example, minute 6 of that video. It sort of reloads the video to have a much shorter length (length changes from 15 minutes to 1, for example) and sometimes I no longer can watch it other times it comes back after about a minute.
I loaded a 15 minute video (/watch?v=u_BcMXgws6Y) and paused right at the start:
Then I jumped into the middle of the video and the seeking/length breaks (length changed to 1:05):
After about a minute it comes back and loads at the correct time, but that is not happening with all videos.
Here's an example with another 15 minute video (/watch?v=9I-34I8OsPM), where the length changes to 1:02 (and then quickly to 1:01), but it does not come back and simply stops loading the video after I do that time jump.
With the workaround I also am not able to change the resolution of a video while playing it.
Weidav commentedon Jul 21, 2025
Yes, if I use the fork, videos won't play at all in Safari due to an error. I also use yattee on mobile devices and my AppleTV and videos won't load there either.
I tried to contact @firejoust but without luck. So might need to fork the fork and create a PR.
Weidav commentedon Mar 1, 2026
I created a solution using systemd that updates the feed every hour. I use Traefik, and for personal/setup reasons, I block the host machine from accessing the Piped API directly. Instead, I run the curl request from the piped-nginx container.
Credits to @0xF4CED I took his original idea and tweaked it to fit my needs.
Store this file in
/usr/local/bin/update_piped_feed.shStore these 2 in
/etc/systemd/system/update_piped_feed.serviceupdate_piped_feed.timerThen activate the service:
EDIT: I originally copied the contend of another systemd .service I use. Now it's the right one. And spelling.