Pagina 1 van 1

How do Spotnet "spots" interact with retention?

Geplaatst: 21 jan 2016, 06:18
door OhneHerren
I'm setting up Spotweb on a Linux server and I'm a bit confused about the underlying technology. It's all working and looking great, but I need some more info to set it up properly. Firstly I'm not sure which provider I should use with it. As far as I understand it currently I can only retrieve as many spots as my Usenet provider can retrieve from the relevant groups. At the moment the oldest spots my provider (UsenetBucket) can find are from 5 years and 11 months ago. If I switch the backend of Spotweb to a provider with greater retention (e.g. my block account with NewsGroupDirect) will I find spots from longer ago than that? Additionally how much bandwidth would a regularly updating spotweb system consume? Currently there's about a 700 day difference between the oldest spot I can see with Spotweb (using UsenetBucket) and the oldest article I could theoretically grab with NewsGroupDirects' retention (provided their retention count has increased linearly since their last announcement).

I also notice that there are different groups for both the "header group" and the "nzb group"? How does this work? Are the "headers" the actual spots? And are the nzbs uploaded to the nzb group at the same time as the spot is uploaded to the header group?

If a Usenet Provider with a longer retention time won't allow me to see older spots or if the bandwidth costs are too high on my block account then I'll stick with just grabbing the relatively younger posts. I wish I could answer these questions myself but I'm having trouble finding much English info on how Spotnet works. Thanks for you help.

Re: How do Spotnet "spots" interact with retention?

Geplaatst: 21 jan 2016, 17:46
door mesa57
There is a special spotweb forum for spotweb related question, but I cannot move you're message.
Anyway :
- Switching to another provider while the database is filled won't add older spots, as spots are retrieved from the newest and so on. So if you want older spots in you're database you can switch to an other provider, but you have to re-initialise you're database.
- It is correct that headers and nzb's are in different groups. For older spots (2011) there are no nzb's present, you have to search an nzbindexer for the filenames. For newer spots the nzb is uploaded to alt.binaries.ftd at the same time the spot is uploaded to free.pt

But be aware : many provider have a long retention for headers and a lower for binaries. That leads to the problem that for older spots you cannot retrieve the nzb anymore.

As to the headers : these are not the complete spots but just an extraction of information in the subject headers of the spots in free.pt. The complete headers+body is retrieved when you're open the spot.

And of course : feel free to ask any other or additional questions :)

Re: How do Spotnet "spots" interact with retention?

Geplaatst: 21 jan 2016, 21:49
door Blauwbek
English support forum is better to ensure better readability for those that don't speak dutch :)

Re: How do Spotnet "spots" interact with retention?

Geplaatst: 22 jan 2016, 00:51
door OhneHerren
Thanks very much for getting back to me mesa57.
mesa57 schreef:- It is correct that headers and nzb's are in different groups. For older spots (2011) there are no nzb's present, you have to search an nzbindexer for the filenames. For newer spots the nzb is uploaded to alt.binaries.ftd at the same time the spot is uploaded to free.pt

But be aware : many provider have a long retention for headers and a lower for binaries. That leads to the problem that for older spots you cannot retrieve the nzb anymore.
Okay so you're saying sometimes I will be able to view the spot but not download the nzb. I have experienced this. It is my understanding that content may be "spotted" at any time after it is uploaded. So it would be possible for there to be a spot I am viewing from three years ago but the nzb file attached to that spot refers to content that is older (for example, five years ago). Is this correct?
mesa57 schreef:As to the headers : these are not the complete spots but just an extraction of information in the subject headers of the spots in free.pt. The complete headers+body is retrieved when you're open the spot.
What do you mean by headers+body? Do you mean the headers are the information about the spot (filesize et cetera) and the body provides is where the nzb file is stored? And where are the comments on a spot stored? Perhaps if you could link me to an example of a "header" and "body" post I could understand better. I'm perhaps not likely to use Spotnet extensively due to the mostly Dutch content it catalogues but I am interested in understanding the technology nonetheless. Maybe there is a detailed Dutch article on how Spotnet works which Google translate could make some sense of?

Oh and also, if I leave a Spotnet system running for a year, how much download bandwidth would it consume to stay up to date?

Thanks again for your help. :)

Re: How do Spotnet "spots" interact with retention?

Geplaatst: 22 jan 2016, 11:30
door mesa57
The NZB can indeed point to files which are even older then the NZB itselfs.
The headers come from the subject of the post and contain all information that is shown in the list.
For details, the whole message had to be read (much slower).
For more technical details you can look at the spotnet source code which is on github. There is also a wiki there explaining to some degree the headers.
Spotnet uses very little bandwidth to load the newest headers as it retrieves only the subject from the messages.
Comments are stored in free.usenet and have a reference to the spot.

Re: How do Spotnet "spots" interact with retention?

Geplaatst: 22 jan 2016, 13:35
door OhneHerren
Ah yes I see Spotweb's database is only 1GB for 5 years of headers.

I think I am coming to some understanding now. That github repo is very useful.

There is indeed a discrepancy between the two Usenet providers I have. The first reports that there is an "Appr. Message count: 2748635" while the second can find "2247994" messages. From this I would expect that the first provider can find older spots than the second, and yet the oldest spots in both cases are the same content. Do you have any idea what this means? Here is an excerpt of the log in both cases.