Samuel Sieb writes:
> On 2021-07-03 8:02 p.m., dwoody5654(a)gmail.com wrote:
> > the url I am trying to download does not have an extension ie. no
> > '.htm' such as:
> >
https://my.acbl.org/club-results/details/338288
The extension doesn't matter to any of the utilities mentioned as far
as I know. I'm pretty sure they get the MIME type from the HTTP
Content-Type header.
> > wget does not download the correct web page.
>
> I tried it and it worked, sort of. The problem is that you want to
> download everything to view it offline, but the site
my.acbl.org has a
> robots.txt that says "no robots allowed". So wget respects that and
> will not download any required files from that site other than the
> initial page. curl probably has the same issue.
1. The page does not have content represented in HTML AFAICT: it's a
blob which is parsed and formatted by a battery of (java)scripts,
some of which are resources on the Internet, and some are inline.
In other words, the HTML in that file is used as a container format
to transport the scripts to the browser.
Neither wget nor curl support Javascript at all as far as I know.
2. 96% of the page is in two blobs; AFAICT there were no IMG or other
elements that specify requirements by URL. If so, that would
explain why only the top page was downloaded.
3. curl does not document how it handles robots.txt. Since as far as
I can tell curl has no recursive or get-requirements option, it
probably doesn't handle it at all.
wget documents that wget -r (recursive downloads) respects
robots.txt. It does not document that wget -p (get page
requisites, too) respects robots.txt, but a quick test suggests
that it does. I think this is a bug: any interactive program that
supports non-text media will download required resources with the
access to the HTML file. (If someone agrees and wants to do
something about it, this is a wget bug, not a Fedora bug.)
I don't have an alternative fetch tool to suggest, unfortunately. I
think that you need to use a graphical browser somehow, or write a
script in your favorite P-language.
Steve
Thanks for the info.
I have been using a script called save-page-as.sh that runs firefox. I have
changed the save-page-as in firefox to use the 'Web Page, complete' The
savd-page-as.sh script sends a ctls-s to firefox and saves the page. It works
perfectly when run from the command line. I have tried to use the
save-page-as.sh script by sending an email to my computer. It does not run
firefox for some reason. In searching it says that firefox can be run from a
cron script by exporting DISPLAY. Running from a cron script, I would
think, is similar to running a script from an email (using procmailrc) no luck
, however.
env shows :0.0. I have tried several variations:
export DISPLAY=:0
export DISPLAY=:0.0
export DISPLAY=:0.1
with no luck.
Perhaps there is another setting that need to be included as well.
Any thoughts?
David
> _______________________________________________
> users mailing list -- users(a)lists.fedoraproject.org
> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
>