Can not import from webpage

Asked by imesg on 2013-10-30

Try to import URL. I get:

"Gourmet does not have a plugin capable of importing URL"

Started in a terminal I get this:

gene@go:~/Docs/tech# gourmet
elib.intl failed to load.
IF YOU HAVE TROUBLE WITH TRANSLATIONS, MAKE SURE YOU HAVE THIS LIBRARY INSTALLED.
elib.intl failed to load.
IF YOU HAVE TROUBLE WITH TRANSLATIONS, MAKE SURE YOU HAVE THIS LIBRARY INSTALLED.
No gst player
No windows player
WARNING: Plugin module import failed
PATH: ['/usr/lib64/python2.7/site-packages/', '/usr/bin', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/gtk-2.0', '/usr/share/gourmet/plugins', '/usr/lib64/python2.7/site-packages/gourmet/plugins', '/usr/lib64/python2.7/site-packages/gourmet/plugins/import_export', '/usr/share/gourmet/plugins/import_export']
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/gourmet/plugin_loader.py", line 252, in get_module
    self._loaded = __import__(self.module)
  File "/usr/lib64/python2.7/site-packages/gourmet/plugins/import_export/web_import_plugin/__init__.py", line 1, in <module>
    import generic_web_importer_plugin
  File "/usr/lib64/python2.7/site-packages/gourmet/plugins/import_export/web_import_plugin/generic_web_importer_plugin.py", line 3, in <module>
    import webpage_importer
  File "/usr/lib64/python2.7/site-packages/gourmet/plugins/import_export/web_import_plugin/webpage_importer.py", line 2, in <module>
    import BeautifulSoup
ImportError: No module named BeautifulSoup
WARNING: Failed to load plugin web_import_plugin
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/gourmet/plugin_loader.py", line 93, in load_active_plugins
    self.active_plugins.extend(self.available_plugin_sets[p].plugins)
  File "/usr/lib64/python2.7/site-packages/gourmet/plugin_loader.py", line 265, in __getattr__
    if attr == 'plugins': return self.get_plugins()
  File "/usr/lib64/python2.7/site-packages/gourmet/plugin_loader.py", line 271, in get_plugins
    return self.get_module().plugins
AttributeError: 'NoneType' object has no attribute 'plugins'
/usr/lib64/python2.7/site-packages/gourmet/recindex.py:122: RuntimeWarning: missing handler 'ingmen_clear'
  'search_as_you_type_toggle' : self.toggleTypeSearchCB,})
/usr/lib64/python2.7/site-packages/gourmet/recindex.py:122: RuntimeWarning: missing handler 'ingmen_pantry'
  'search_as_you_type_toggle' : self.toggleTypeSearchCB,})
/usr/lib64/python2.7/site-packages/gourmet/recindex.py:122: RuntimeWarning: missing handler 'print'
  'search_as_you_type_toggle' : self.toggleTypeSearchCB,})
/usr/lib64/python2.7/site-packages/gourmet/recindex.py:122: RuntimeWarning: missing handler 'email'
  'search_as_you_type_toggle' : self.toggleTypeSearchCB,})
/usr/lib64/python2.7/site-packages/gourmet/recindex.py:122: RuntimeWarning: missing handler 'batch_edit'
  'search_as_you_type_toggle' : self.toggleTypeSearchCB,})
/usr/lib64/python2.7/site-packages/gourmet/recindex.py:122: RuntimeWarning: missing handler 'rl_delrec'
  'search_as_you_type_toggle' : self.toggleTypeSearchCB,})
/usr/lib64/python2.7/site-packages/gourmet/recindex.py:122: RuntimeWarning: missing handler 'rl_viewrec'
  'search_as_you_type_toggle' : self.toggleTypeSearchCB,})
/usr/lib64/python2.7/site-packages/gourmet/recindex.py:122: RuntimeWarning: missing handler 'rl_shoprec'
  'search_as_you_type_toggle' : self.toggleTypeSearchCB,})
CONTENT TYPE = text/html; charset=utf-8
emit ('completed',)
emit ('done',)
/usr/lib64/python2.7/site-packages/gourmet/threadManager.py:307: GtkWarning: IA__gtk_progress_set_percentage: assertion `percentage >= 0 && percentage <= 1.0' failed
  pb.set_fraction(perc)

What goes on here?

Thanks,

Question information

Language:
English Edit question
Status:
Solved
For:
Gourmet Edit question
Assignee:
No assignee Edit question
Solved by:
imesg
Solved:
2013-11-10
Last query:
2013-11-10
Last reply:
2013-11-06
Bernhard Reiter (ockham-razor) said : #1

Seems like you don't have BeautifulSoup installed, which is required for website parsing.

To install it under Debian/Ubuntu, run
sudo apt-get install python-beautifulsoup
from a terminal.

If you're on another Linux distribution or operating system, tell me.

imesg (imese) said : #2

You are exactly right about beautifulsoup. However I don't know if this will fix the problem because gentoo is in the middle of a big fubar with python-exec and I can't install anything related to python (bummer). As soon as this fubar sorts itself I will report back.

Most likely the problem is that the gentoo ebuild does not include beautifulsoup as a dependency for gourmet. Will you see to this or should I post a bug report with gentoo?

Thanks for the help as always.

Great program by the way.

Gene

Bernhard Reiter (ockham-razor) said : #3

Am 2013-11-03 18:31, schrieb imesg:
> Question #238352 on Gourmet changed:
> https://answers.launchpad.net/gourmet/+question/238352
>
> imesg posted a new comment:
> You are exactly right about beautifulsoup. However I don't know if this
> will fix the problem because gentoo is in the middle of a big fubar with
> python-exec and I can't install anything related to python (bummer). As
> soon as this fubar sorts itself I will report back.
>
> Most likely the problem is that the gentoo ebuild does not include
> beautifulsoup as a dependency for gourmet. Will you see to this or
> should I post a bug report with gentoo?
I've taken a look at the current unstable 0.16.1 ebuild
(http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/app-misc/gourmet/)
and, having been a Gentoo user myself a long time ago, attempted to fix it.
This includes updates to other RDEPENDencies and USE flags as well, see
attached patch. (Note that after applying that patch,
gourmet-0.16.1.ebuild still needs to be renamed to
gourmet-0.16.1-r1.ebuild.) Also, the change to manifest.xml probably
requires updating the USE flags for the 0.15.9 ebuild found at the same
location.

I'm CC'ing other potentially interested parties. Please note that I'm
not a Gentoo developer nor user these days, so *I did not test this
ebuild*. I still hope it's useful and makes its way to the Gentoo
repositories in some way or another -- please keep me posted!
> Thanks for the help as always.
You're welcome, thanks for telling me about this bug.
> Great program by the way.
Kudos go to Tom Hinkle, who started it almost some 10 years ago -- I've
been maintaining it only relatively recently.
> Gene
Bernhard

Bernhard Reiter (ockham-razor) said : #4

(Find the full message including the patch at https://lists.launchpad.net/gourmet/msg00007.html)

Joe Sapp (sappj) said : #5

I'm the current maintainer of this package and I was working on an improved ebuild. I had been waiting to get elib.intl into the tree first, but I saw some more pressing issues that needed to be addressed -- especially when you pointed out this bug. Enable "web" in your USE variable and update to 0.16.1-r1.

Bernhard Reiter (ockham-razor) said : #6

Am 2013-11-05 15:41, schrieb Joe Sapp:
> Question #238352 on Gourmet changed:
> https://answers.launchpad.net/gourmet/+question/238352
>
> Joe Sapp posted a new comment:
> I'm the current maintainer of this package and I was working on an
> improved ebuild. I had been waiting to get elib.intl into the tree
> first, but I saw some more pressing issues that needed to be addressed
> -- especially when you pointed out this bug. Enable "web" in your USE
> variable and update to 0.16.1-r1.

Just noticed
http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/app-misc/gourmet/gourmet-0.16.1-r1.ebuild?view=markup
-- thanks!

A couple of notes:
* Please drop the gnome-print flag. Those dependencies are obsolete now,
as we don't rely on the legacy Gnome printing framework anymore. The
only dependencies required for printing these days are reportlab and
python-poppler, i.e. the same as for the pdf flag. (The "Gnome part" of
printing -- such as the print dialog -- has long been integrated with Gtk+.)
* For the web flag, please rdepend on the python-2 slot of beautifulsoup
(i.e. on dev-python/beautifulsoup:python-2) (which amounts to
beautifulsoup version 3.x, excluding 4.x versions, which currently won't
work.)
* PyGTK >= 2.3.96 isn't sufficient anymore these days; OTOH, any Gtk+
3.x versions (from the "3" slot) won't work, so you should depend on
>=dev-python/pygtk-2.22.0:2
* The old metakit database backend isn't supported anymore, so you can
drop the dev-db/metakit[python] dependency. (Legacy files still present
in Gourmet's source tree will be dropped in the next release).
* RTF support isn't currently supported, so you can drop that flag too.
* Please depend on >=dev-python/sqlalchemy-0.7.0, as we require the 0.7
API (or later) these days.
* For sound (i.e. the timer alarm), you need dev-python/gst-python
* For the interactive python console plugin, you need dev-python/ipython

These requirements might change in the course of new releases (e.g. we
are going to switch to BeautifulSoup 4 with 0.17 or so), which I try to
document in https://github.com/thinkle/gourmet/blob/master/INSTALL.md,
so you can just check out that file when there's a new release.

Kind regards
Bernhard

imesg (imese) said : #7

Thanks, Bernhard. I finally got the import to work except for importing a png file i.e.

WARNING: Error on creating thumbnail - ignoring
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/gourmet/importers/imageBrowser.py", line 30, in grab_thumbnail
    fi = check_for_thumbnail(uri,type,reporthook)
  File "/usr/lib64/python2.7/site-packages/gourmet/gtk_extras/thumbnail.py", line 41, in check_for_thumbnail
    return create_thumbnail(fn,name,uri,type)
  File "/usr/lib64/python2.7/site-packages/gourmet/gtk_extras/thumbnail.py", line 84, in create_thumbnail
    import PngImagePlugin
ImportError: No module named PngImagePlugin

I think this may be a problem with having python2 and python3 installed at the same time. I will have to work on this one.

Also the 'web' USE flag did not necessarily pull in beautifulsoup. I had to reemerge --deep world to get python to pull it in.

In short I can import but not with images.

Also again, if I want to emerge ipython it also pulls in gnome-print etc. Not your problem though I don't think.

Brings up another question though. When will you migrate to python3?

Thanks for your help, I thinks we can call this solved.

Gene

Bernhard Reiter (ockham-razor) said : #8

Am 2013-11-10 19:06, schrieb imesg:
> Thanks, Bernhard. I finally got the import to work except for importing
> a png file i.e.
>
> WARNING: Error on creating thumbnail - ignoring
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/site-packages/gourmet/importers/imageBrowser.py", line 30, in grab_thumbnail
> fi = check_for_thumbnail(uri,type,reporthook)
> File "/usr/lib64/python2.7/site-packages/gourmet/gtk_extras/thumbnail.py", line 41, in check_for_thumbnail
> return create_thumbnail(fn,name,uri,type)
> File "/usr/lib64/python2.7/site-packages/gourmet/gtk_extras/thumbnail.py", line 84, in create_thumbnail
> import PngImagePlugin
> ImportError: No module named PngImagePlugin
>
> I think this may be a problem with having python2 and python3 installed
> at the same time. I will have to work on this one.

It was actually due to the python-imaging -> PIL migration that is happening in most distributions, and that is remotely related to the py2 -> py3 migration. Anyway, thanks for pointing it out -- now fixed by https://github.com/thinkle/gourmet/commit/4eb110bbbc58af97fd6994b96d9482783037058d

> Also the 'web' USE flag did not necessarily pull in beautifulsoup. I had
> to reemerge --deep world to get python to pull it in.

I'm afraid I can't help much further with Gentoo related issues. But in case you missed my other message to Joe Sapp, the maintainer of the Gourmet ebuild, see https://lists.launchpad.net/gourmet/msg00008.html -- I'm requesting a couple of changes to 0.16.1-r1 there.

> In short I can import but not with images.
>
> Also again, if I want to emerge ipython it also pulls in gnome-print
> etc. Not your problem though I don't think.
>
> Brings up another question though. When will you migrate to python3?

See https://github.com/thinkle/gourmet/issues/697 -- it's somewhat involved as we need to migrate from PyGTK 2.x to PyGobject/GTK 3 first, which isn't available on Windows yet.

> Thanks for your help, I thinks we can call this solved.

You're welcome.

Bernhard

Bernhard Reiter (ockham-razor) said : #9

Am 2013-11-10 19:06, schrieb imesg:
> Thanks, Bernhard. I finally got the import to work except for importing
> a png file i.e.
>
> WARNING: Error on creating thumbnail - ignoring
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/site-packages/gourmet/importers/imageBrowser.py", line 30, in grab_thumbnail
> fi = check_for_thumbnail(uri,type,reporthook)
> File "/usr/lib64/python2.7/site-packages/gourmet/gtk_extras/thumbnail.py", line 41, in check_for_thumbnail
> return create_thumbnail(fn,name,uri,type)
> File "/usr/lib64/python2.7/site-packages/gourmet/gtk_extras/thumbnail.py", line 84, in create_thumbnail
> import PngImagePlugin
> ImportError: No module named PngImagePlugin
>
> I think this may be a problem with having python2 and python3 installed
> at the same time. I will have to work on this one.
It was actually due to the python-imaging -> PIL migration that is
happening in most distributions, and that is remotely related to the py2
-> py3 migration. Anyway, thanks for pointing it out -- now fixed by
https://github.com/thinkle/gourmet/commit/4eb110bbbc58af97fd6994b96d9482783037058d
> Also the 'web' USE flag did not necessarily pull in beautifulsoup. I had
> to reemerge --deep world to get python to pull it in.
I'm afraid I can't help much further with Gentoo related issues. But in
case you missed my other message to Joe Sapp, the maintainer of the
Gourmet ebuild, see https://lists.launchpad.net/gourmet/msg00008.html --
I'm requesting a couple of changes to 0.16.1-r1 there.
(CC'ing Joe to ping him)
> In short I can import but not with images.
>
> Also again, if I want to emerge ipython it also pulls in gnome-print
> etc. Not your problem though I don't think.
>
> Brings up another question though. When will you migrate to python3?
See https://github.com/thinkle/gourmet/issues/697 -- it's somewhat
involved as we need to migrate from PyGTK 2.x to PyGobject/GTK 3 first,
which isn't available on Windows yet.
> Thanks for your help, I thinks we can call this solved.
You're welcome.

Bernhard

Joe Sapp (sappj) said : #10

An ebuild incorporating these improvements should be hitting the mirrors soon. Thanks for the info, Bernhard, and thanks for the report imese.