Remote lib retrieval for eeschema (like pcbnew)

I will be glad to contribute to your roadmap.
Am specially interested on helping adding a MySQL plugin like if it is willing to be in next versions.

Thanks Wayne Stambaugh, that solved my question.

We have discussed the possibility of creating a database plugin in the
past. It has never gone anywhere other than the discussion phase. I'm
comfortable with the idea. I would like to see it be more generic like
a SQL database plugin so as to make it useful for any database the
supports SQL. If you tie directily to MySQL, it adds a direct
dependency to MySQL and limits it's usefulness. My guess is users will
prefer to use there own database front end be it MySQL, PostgreSQL,
Oracle, etc.

On 8/8/2017 11:14 AM, hackinvent Badr wrote:
> Question #655449 on KiCad changed:
> https://answers.launchpad.net/kicad/+question/655449
>
> hackinvent Badr posted a new comment:
> I will be glad to contribute to your roadmap.
> Am specially interested on helping adding a MySQL plugin like if it is willing to be in next versions.
>

If the interface is generic enough then it may also allow us to query websites for data.

@Badr, I think you should join the developer mailing list [1], so we could discuss the idea further. Perhaps you will find the a few clever ideas or developers willing to help you.

1. http://launchpad.net/~kicad-developers

@Maciej thank you, just requested subscription

@Stambaugh am not specially tied to MySQL but just because we use it in our internal webservices, I will make some experimentations locally within the next days to learn a bit about Kicad source, and try to connect to our MySQL database where we store some kicad components at Hackivent

Can you please indicate me where can I read about Kicad policy regarding using 3rd part. libraries (ex. Qt5 libs)?

On 8/8/2017 11:53 AM, hackinvent Badr wrote:
> Question #655449 on KiCad changed:
> https://answers.launchpad.net/kicad/+question/655449
>
> hackinvent Badr posted a new comment:
> @Maciej thank you, just requested subscription
>
> @Stambaugh am not specially tied to MySQL but just because we use it in
> our internal webservices, I will make some experimentations locally
> within the next days to learn a bit about Kicad source, and try to
> connect to our MySQL database where we store some kicad components at
> Hackivent
>
> Can you please indicate me where can I read about Kicad policy regarding
> using 3rd part. libraries (ex. Qt5 libs)?
>

We don't have a written policy on library dependencies but there is an
unwritten rule that any new library must be readily available (packaged
for some type of installer) on the three major platforms, windows, osx,
and linux. If a library is not available, then it becomes the
responsibility of the developer to provide packages or at least the
package build source. We have done this a few times primarily on
windows but it's not very common. I would avoid QT since we currently
use wxWidgets for our UI code. Using competing UI libraries will only
make our UI more inconsistent than it already is.

I see. Personnaly if I use Qt it would be only for its core+sql libs not for the GUI.
It has many non-gui libs that handle databases, web requests, json parsers,...etc that make dev more easier.

On 8/8/2017 12:14 PM, hackinvent Badr wrote:
> Question #655449 on KiCad changed:
> https://answers.launchpad.net/kicad/+question/655449
>
> hackinvent Badr posted a new comment:
> I see. Personnaly if I use Qt it would be only for its core+sql libs not for the GUI.
> It has many non-gui libs that handle databases, web requests, json parsers,...etc that make dev more easier.
>

Boost provides a json parser. wxWidgets (and possibly Boost) can handle
web requests. I'm not aware of any database code for Boost or wxWidgets
but it may already exist. I would prefer to use existing dependencies
if at all possible.

I just finished looking at the process of the update/instert of library in eeschema (rev. 4.0.7).

Maybe I am wrong, what I understood is that when I make a modification on a library, and after an independant event (update on the sch screen for ex) the existing components in the SCH_COMPONENT::cmp_list resolve their parts from the lib directly from the files.

This approache maybe discuraging if we want to modify it to implement database or remote file links.

What I would do to start is to create in a first step a local SCH_COMPONENT::OrderedTreeHashMap<name or id of lib, pointer to lib's object> that stores locally all the parts from the configured libraries.
It is filled when eeschema starts and updated when you click on ok in the DIALOG_EESCHEMA_CONFIG.

When a component wants to resorlve the part list it directly looks in the map of lib parts. Which will be a fast search.

This map in a second step can be filled from remote libs and database easily.

I don't know if someone has explored this approche? And since I didn't look deeped in the code, I don't know if there will some side effects?

Erratum PART_LIBS:::OrderedTreeHashMap<name or id of lib, pointer to lib's object>
Précision, the map will be static in PART_LIBS class.

In another hand, regarding the configuration of the remote access.
Today the content of *.lib files is: EESchema-LIBRARY Version 2.3 .....

The idea to limit the code/architecture modification is we can imaging the following types of lib that contain the configuration parameters:
EESchema-GITHUB Version 1.0
https://git.example.com/...
...
----------------------
EESchema-URI Version 1.0
https://www.example.com/mylib.lib
...
----------------------
EESchema-DATABASE Version 1.0
mysql://loging:<email address hidden>:port/database/table
postgre://loging:<email address hidden>:port/database/table
....
----------------------

When the configuration lib is loaded, we retrieve by case the type of remote library loading.
In a first step configuration files would be manually modified, then we can imagin a dialog window that can create them locally.

I think I got some early conclusions, it is more complicated (still noob on the subject) than I expected.

The first step will be (you may not like it) is to modify the SCH_LEGACY_PLUGIN_CACHE::Load()... it will be quick and some how dirty. A legacy with remote retrieval support, that's no more legacy.

The modifications will be the following:
- 2 new types of libs: EESchema-URI et EESchema-DATABASE
- EESchema-URI will contain the list of remote lib files (type EESchema-LIB), with check to avoid recursivity in the ::load() method.
- EESchema-DATABASE will contain the list of remote MySQL tables that contains EESchema-LIB data in each row

- In the ::load() method we check what is the type of the lib, when it is
--> EESchema-LIB the process is already coded
--> EESchema-URI, it will get the remote file and process it as EESchema-LIB
--> EESchema-DATABASE, it will connect to the remote data base, retrieve all the rows, and process the content of each row as EESchema-LIB

In the first step, I will try to make work remote lib retrieval function EESchema-URI.
The coding of the retrieval of the remote libs will be done as following:
- implement WEB_FILE_LINE_READER in richio similar to FILE_LINE_READER based on the wxURL/wxInputStream/INPUTSTREAM_LINE_READER...
- add new method SCH_LEGACY_PLUGIN_CACHE::LoadRemote(wxString url)
--> it will retmotly get the file using WEB_FILE_LINE_READER
--> it will will check that the file is EESchema-LIB, else no processing
--> the parts parsing we be done as done today by ::load() method

This way less effor without side effects.

I will give it a try this weekend.
If things go ok I will move to database implementation.

Erratum: Since wsURL doesn't support ssl (https), I will rather use kicad_curl_easy as done for pcbnew in GITHUB_PLUGIN::FootprintLoad

On 8/11/2017 9:44 AM, hackinvent Badr wrote:
> Question #655449 on KiCad changed:
> https://answers.launchpad.net/kicad/+question/655449
>
> hackinvent Badr posted a new comment:
> I think I got some early conclusions, it is more complicated (still noob
> on the subject) than I expected.
>
> The first step will be (you may not like it) is to modify the
> SCH_LEGACY_PLUGIN_CACHE::Load()... it will be quick and some how dirty.
> A legacy with remote retrieval support, that's no more legacy.
>
> The modifications will be the following:
> - 2 new types of libs: EESchema-URI et EESchema-DATABASE
> - EESchema-URI will contain the list of remote lib files (type EESchema-LIB), with check to avoid recursivity in the ::load() method.
> - EESchema-DATABASE will contain the list of remote MySQL tables that contains EESchema-LIB data in each row
>
> - In the ::load() method we check what is the type of the lib, when it is
> --> EESchema-LIB the process is already coded
> --> EESchema-URI, it will get the remote file and process it as EESchema-LIB
> --> EESchema-DATABASE, it will connect to the remote data base, retrieve all the rows, and process the content of each row as EESchema-LIB
>
> In the first step, I will try to make work remote lib retrieval function EESchema-URI.
> The coding of the retrieval of the remote libs will be done as following:
> - implement WEB_FILE_LINE_READER in richio similar to FILE_LINE_READER based on the wxURL/wxInputStream/INPUTSTREAM_LINE_READER...
> - add new method SCH_LEGACY_PLUGIN_CACHE::LoadRemote(wxString url)
> --> it will retmotly get the file using WEB_FILE_LINE_READER
> --> it will will check that the file is EESchema-LIB, else no processing
> --> the parts parsing we be done as done today by ::load() method
>
> This way less effor without side effects.
>
> I will give it a try this weekend.
> If things go ok I will move to database implementation.
>

You should not need to change the SCH_PLUGIN interface. Just create a
new plugin to handle legacy files from a different source. Load() and
Save() are for loading and saving schematics. This has nothing to do
with symbol library loading and saving. The symbol libraries are
handled in the XXXXLib() functions. Plugins only need to implement a
subset of the plugin API as applicable (ie read only like the github
plugin). Use properties if your plugin requires additional information
rather than change the SCH_PLUGIN API. What you do inside your plugin
is completely up to you but the SCH_PLUGIN interface should not change.

On 8/10/2017 7:47 PM, hackinvent Badr wrote:
> Question #655449 on KiCad changed:
> https://answers.launchpad.net/kicad/+question/655449
>
> hackinvent Badr posted a new comment:
> I just finished looking at the process of the update/instert of library
> in eeschema (rev. 4.0.7).
>
> Maybe I am wrong, what I understood is that when I make a modification
> on a library, and after an independant event (update on the sch screen
> for ex) the existing components in the SCH_COMPONENT::cmp_list resolve
> their parts from the lib directly from the files.

This is going to change soon. The SYMBOL_LIB_TABLE method for looking
up symbol libraries is currently being written by me. This will connect
the appropriate SCH_PLUGIN to the proper symbol library. The current
method will only be used for importing legacy schematics and remapping
the library symbol links over to the SYMBOL_LIB_TABLE.

>
> This approache maybe discuraging if we want to modify it to implement
> database or remote file links.

This method of linking symbols in the schematic to libraries is going to
go away in the new schematic file format. In the short term, we will
have to live with this. Symbols will be embedded in the new schematic
file format in the same way that footprints are embedded in the board
file format.

>
> What I would do to start is to create in a first step a local SCH_COMPONENT::OrderedTreeHashMap<name or id of lib, pointer to lib's object> that stores locally all the parts from the configured libraries.
> It is filled when eeschema starts and updated when you click on ok in the DIALOG_EESCHEMA_CONFIG.
>
> When a component wants to resorlve the part list it directly looks in
> the map of lib parts. Which will be a fast search.
>
> This map in a second step can be filled from remote libs and database
> easily.
>
> I don't know if someone has explored this approche? And since I didn't
> look deeped in the code, I don't know if there will some side effects?
>

This discussion should really be moved to the developers mailing list
rather than as a question.

Yes, I will send a message in the mailing list.
I just finished the update, it is working without side effect in the existing behavior, I will share the modification.
Thanks