Priority of paths in Library Preferences not OK

Well, Rob, the bottom field is still having a prioritive meaning, and it also works as intended. But none of the modules or components within such an additional library are read when the path is added. Indeed individual library files from such a path must be added in the top field. However, it seems that the top field has its own priorities which make the path priority meaningless?

The cache library is never at the top of the priority list. You can put it anywhere you want but internally Eeschema will always move it to the end of the search list. This is by design. The cache is only used as a fail safe when a component cannot be found in the list of libraries. That is why when you send it to someone who doesn't have your custom libraries it works because Eeschema is using the cache library. Your problem stems from a known design issue with Eeschema that has been documented many times. Eeschema does not save the library name when you add a new component. When schematic is loaded the next time, Eeschema will use the first component having the same name as the component that was added which may or may not be the component that was actually added to the schematic. This problem will be address when the new file format is implemented. In the mean time, here are a few simple rules to prevent this from occurring:

1) Always use unique part names even if you copied and modified an existing part. Duplicate part names cause the problem. I always suffix my custom component names with my initials.

2) Do not modify the stock Kicad libraries. No one else is likely to have these changes in their libraries.

3) Create a new library in your project path and put all your custom components into it and add it to the beginning of the library search list. This way your project will be relocatable. If your custom library is not in the project path, you will have to remember to copy it to each system in the correct path which can cause problems if you work on different platforms.

Follow these few simple rules and you shouldn't have any problems.

> The cache library is never at the top of the priority list. You can put
> it anywhere you want but internally Eeschema will always move it to the
> end of the search list. This is by design. The cache is only used as a

Actually, when the cache is put on top, eeschema also behaves
appropriately. So, it is NOT internally moved to the end of the list.

> fail safe when a component cannot be found in the list of libraries.
> That is why when you send it to someone who doesn't have your custom
> libraries it works because Eeschema is using the cache library.

So far, this is correct.

> Your
> problem stems from a known design issue with Eeschema that has been
> documented many times. Eeschema does not save the library name when you
> add a new component. When schematic is loaded the next time, Eeschema
> will use the first component having the same name as the component that
> was added which may or may not be the component that was actually added
> to the schematic.

It is this, where the paths priorities apparently get in.

> 2) Do not modify the stock Kicad libraries. No one else is likely to
> have these changes in their libraries.

Well, here lies a potential problem. In certain time intervals, the stock
libraries are updated, mostly because of new parts, but probably also
because of improved parts. Well, here a schematic that is handed over for
review would get 'corrupted' when the companion has another revision
version of the stock libraries. This problem would also exist when the
library name is ever saved when I add a new component. But let me keep the
discussion generic. When the behaviour as you describe it is by design,
than effectively a KiCad design behaves like a relational database. Upon
opening a schematic, components are nominally always loaded from the
libraries. Contrary to this the board file: here you MUST open a dialog
when you want to update a footprint from an updated footprint library. To
my opinion, the same concept should apply to schematics. I remember from
Orcad, that you could select between cache or libraries, if you would add
a new component that is already in the cache. But upon opening a
schematic, always the cache is taken.

BTW: I updated footprint updates in the stock libraries, e.g. erratic
SOT-23. Doing so is currently experimental, namely accompanied by keeping
track of library changes via a git repository (it is this background why I
reported a 'bug' (wish point) regarding the storage of modified footprints
at the tail of the mod file).

> 3) Create a new library in your project path and put all your custom
> components into it and add it to the beginning of the library search
> list. This way your project will be relocatable.

This is actually the same method as I'd practice with the cache.

On 12/6/2011 10:30 AM, Rob Maris wrote:
> Question #181052 on KiCad changed:
> https://answers.launchpad.net/kicad/+question/181052
>
> Status: Answered => Open
>
> Rob Maris is still having a problem:
>> The cache library is never at the top of the priority list. You can put
>> it anywhere you want but internally Eeschema will always move it to the
>> end of the search list. This is by design. The cache is only used as a
>
> Actually, when the cache is put on top, eeschema also behaves
> appropriately. So, it is NOT internally moved to the end of the list.

This code would suggest otherwise:

wxArrayString CMP_LIBRARY::GetLibraryNames( bool aSorted )
{
    wxString cacheName;
    wxArrayString names;

    BOOST_FOREACH( CMP_LIBRARY& lib, CMP_LIBRARY::libraryList )
    {
        if( lib.isCache && aSorted )
            cacheName = lib.GetName();
        else
            names.Add( lib.GetName() );
    }

    /* Even sorted, the cache library is always at the end of the list. */
    if( aSorted )
        names.Sort();

    if( !cacheName.IsEmpty() )
        names.Add( cacheName );

    return names;
}

I wrote this code which emulates the original behavior of the code this
replaced so I have a pretty good handle on library search behavior.

>
>> fail safe when a component cannot be found in the list of libraries.
>> That is why when you send it to someone who doesn't have your custom
>> libraries it works because Eeschema is using the cache library.
>
> So far, this is correct.
>
>> Your
>> problem stems from a known design issue with Eeschema that has been
>> documented many times. Eeschema does not save the library name when you
>> add a new component. When schematic is loaded the next time, Eeschema
>> will use the first component having the same name as the component that
>> was added which may or may not be the component that was actually added
>> to the schematic.
>
> It is this, where the paths priorities apparently get in.
>
>> 2) Do not modify the stock Kicad libraries. No one else is likely to
>> have these changes in their libraries.
>
> Well, here lies a potential problem. In certain time intervals, the stock
> libraries are updated, mostly because of new parts, but probably also
> because of improved parts. Well, here a schematic that is handed over for
> review would get 'corrupted' when the companion has another revision
> version of the stock libraries. This problem would also exist when the
> library name is ever saved when I add a new component. But let me keep the
> discussion generic. When the behaviour as you describe it is by design,
> than effectively a KiCad design behaves like a relational database. Upon
> opening a schematic, components are nominally always loaded from the
> libraries. Contrary to this the board file: here you MUST open a dialog
> when you want to update a footprint from an updated footprint library. To
> my opinion, the same concept should apply to schematics. I remember from
> Orcad, that you could select between cache or libraries, if you would add
> a new component that is already in the cache. But upon opening a
> schematic, always the cache is taken.
>
> BTW: I updated footprint updates in the stock libraries, e.g. erratic
> SOT-23. Doing so is currently experimental, namely accompanied by keeping
> track of library changes via a git repository (it is this background why I
> reported a 'bug' (wish point) regarding the storage of modified footprints
> at the tail of the mod file).
>
>> 3) Create a new library in your project path and put all your custom
>> components into it and add it to the beginning of the library search
>> list. This way your project will be relocatable.
>
> This is actually the same method as I'd practice with the cache.

You should never directly modify the cache with the library editor. Your
changes will be completely overwritten the next time your schematic is saved.
The cache is a snapshot of the components in the schematic at the last point
the schematic was saved. You must create a new library and add your components
there.

>> Actually, when the cache is put on top, eeschema also behaves
>> appropriately. So, it is NOT internally moved to the end of the list.
>
> This code would suggest otherwise:
>
> wxArrayString CMP_LIBRARY::GetLibraryNames( bool aSorted )
> {
> wxString cacheName;
> wxArrayString names;
>
> BOOST_FOREACH( CMP_LIBRARY& lib, CMP_LIBRARY::libraryList )
> {
> if( lib.isCache && aSorted )
> cacheName = lib.GetName();
> else
> names.Add( lib.GetName() );
> }
>
> /* Even sorted, the cache library is always at the end of the list.
> */
> if( aSorted )
> names.Sort();
>
> if( !cacheName.IsEmpty() )
> names.Add( cacheName );
>
> return names;
> }
>
> I wrote this code which emulates the original behavior of the code this
> replaced so I have a pretty good handle on library search behavior.

I have done following tests (using KiCad BZR 3249):
- I make a change in a component from the standard libs, *after* it has
been inserted in the schematic before in a separate session. Hence, the
cache holds the old version, the libs the altered version.
- when this component change is saved, you can see it immediately in the
schematic (i.e. behaviour a la relational database). Yet it is not written
to the cache.
- then I add xxx-cache.lib on top of the lib preferences dialog "Component
library files" and the new version is still visible.
- I exit and restart eeschema: the cache version is visible.
- when I remove xxx-cache.lib from the list, after hitting the OK button,
the changed version is visible immediately.

Another problem: when I send a schematic to a companion who doesn't have
my own custom libs, the system will complain missing libs, because the
file names that I use in a particular design are listed in the .pro file.
But this I treat harmless. More important is correct rendering.

>>> 3) Create a new library in your project path and put all your custom
>>> components into it and add it to the beginning of the library search
>>> list. This way your project will be relocatable.
>>
>> This is actually the same method as I'd practice with the cache.
>
> You should never directly modify the cache with the library editor. Your
> changes will be completely overwritten the next time your schematic is
> saved.

No no, I never edit the cache. When I say "the same practice" I only meant
to say: put a library on top of the list!

BTW: I highly appreciate the quick and extensive answering.

-Rob

On 12/6/2011 2:00 PM, Rob Maris wrote:
> Question #181052 on KiCad changed:
> https://answers.launchpad.net/kicad/+question/181052
>
> Status: Answered => Open
>
> Rob Maris is still having a problem:
>>> Actually, when the cache is put on top, eeschema also behaves
>>> appropriately. So, it is NOT internally moved to the end of the list.
>>
>> This code would suggest otherwise:
>>
>> wxArrayString CMP_LIBRARY::GetLibraryNames( bool aSorted )
>> {
>> wxString cacheName;
>> wxArrayString names;
>>
>> BOOST_FOREACH( CMP_LIBRARY& lib, CMP_LIBRARY::libraryList )
>> {
>> if( lib.isCache && aSorted )
>> cacheName = lib.GetName();
>> else
>> names.Add( lib.GetName() );
>> }
>>
>> /* Even sorted, the cache library is always at the end of the list.
>> */
>> if( aSorted )
>> names.Sort();
>>
>> if( !cacheName.IsEmpty() )
>> names.Add( cacheName );
>>
>> return names;
>> }
>>
>> I wrote this code which emulates the original behavior of the code this
>> replaced so I have a pretty good handle on library search behavior.
>
> I have done following tests (using KiCad BZR 3249):
> - I make a change in a component from the standard libs, *after* it has
> been inserted in the schematic before in a separate session. Hence, the
> cache holds the old version, the libs the altered version.
> - when this component change is saved, you can see it immediately in the
> schematic (i.e. behaviour a la relational database). Yet it is not written
> to the cache.

The cache write doesn't happen until you save the schematic. It is not updated
automatically if a library component is changed. So if you don't save the
schematic, the cache will not be updated.

> - then I add xxx-cache.lib on top of the lib preferences dialog "Component
> library files" and the new version is still visible.

Because you really didn't add it to the beginning of the search list (see code
above) it only appeared that way in the library search order dialog so it was
still using the part in your custom library. The library dialog probably
should remove the cache from the sort order list to prevent confusion. It is
possible there is a bug where the dialog changes the sort order without calling
the code above causing the sort order to be invalid until the next time
Eeschema is restarted. I will take a look at it when I get a chance because
this shouldn't happen. If it is happening I will file a bug report and fix it.

> - I exit and restart eeschema: the cache version is visible.
> - when I remove xxx-cache.lib from the list, after hitting the OK button,
> the changed version is visible immediately.

This is expected if you didn't save the schematic to regenerate the cache. You
probably shouldn't be able to delete the cache library either. It will be
regenerated when you save your schematic and automatically added to the end of
the list the next time you open the schematic.

> Another problem: when I send a schematic to a companion who doesn't have
> my own custom libs, the system will complain missing libs, because the
> file names that I use in a particular design are listed in the .pro file.
> But this I treat harmless. More important is correct rendering.
>
>
>>>> 3) Create a new library in your project path and put all your custom
>>>> components into it and add it to the beginning of the library search
>>>> list. This way your project will be relocatable.
>>>
>>> This is actually the same method as I'd practice with the cache.
>>
>> You should never directly modify the cache with the library editor. Your
>> changes will be completely overwritten the next time your schematic is
>> saved.
>
> No no, I never edit the cache. When I say "the same practice" I only meant
> to say: put a library on top of the list!

Oops! I misunderstood what you were trying to say. Thank you for your step by
step explanation. It will make it easier to find the problem.

Wayne

>
> BTW: I highly appreciate the quick and extensive answering.
>
> -Rob
>

>> - I make a change in a component from the standard libs, *after* it has
>> been inserted in the schematic before in a separate session. Hence, the
>> cache holds the old version, the libs the altered version.
>> - when this component change is saved, you can see it immediately in the
>> schematic (i.e. behaviour a la relational database). Yet it is not
>> written to the cache.
>
> The cache write doesn't happen until you save the schematic. It is not
> updated automatically if a library component is changed. So if you
> don't save the schematic, the cache will not be updated.

Yes, I'm aware of that. I'm only describing what I'm obeying. In this case
that a change in a library is immediately reflected in an open design
(which is undesired - to my opinion).

>> - then I add xxx-cache.lib on top of the lib preferences dialog
>> "Component library files" and the new version is still visible.
>
> Because you really didn't add it to the beginning of the search list
> (see code
> above) it only appeared that way in the library search order dialog so
> it was

Yes, there is some asymmetry in the system's 'awareness' of changes: lib
changes immediately; changes of search order after schematic reopen (the
latter I expected).

> still using the part in your custom library. The library dialog probably
> should remove the cache from the sort order list to prevent confusion.
> It is possible there is a bug where the dialog changes the sort order
> without
> calling the code above causing the sort order to be invalid until the
> next time
> Eeschema is restarted. I will take a look at it when I get a chance
> because
> this shouldn't happen. If it is happening I will file a bug report and
> fix it.
>
>> - I exit and restart eeschema: the cache version is visible.
>> - when I remove xxx-cache.lib from the list, after hitting the OK
>> button,
>> the changed version is visible immediately.
>
> This is expected if you didn't save the schematic to regenerate the
> cache. You
> probably shouldn't be able to delete the cache library either. It will
> be

Note that I do not delete the cache library, I removed its entry in the
prirority list and for the test I intentionally wanted to exploit what is
happening when there is a difference.

> regenerated when you save your schematic and automatically added to the
> end of the list the next time you open the schematic.

Indeed, I did not save the schematic. When I would do that before exiting,
then of course there would be no difference when I remove xxx-cache.lib
 from the priority list, because lib and cache are identical (regarding
that sample component). However: what I believe to have proved is: the
cache has priority over standard libs when I intentionally put the cache
file on top of the list. So to my opinion, this priority is not
'automatically' destroyed by the algorithm (but perhaps there is some
commutal misunderstanding...). Consider the following: I improve a
schematic symbol in the standard libs, while NO schematic project is open
(at least of project xyz). When I later open xyz, I'd find updated
schematic symbols. However, when the cache would be entered into the list
(on top), this would not happen (I know, currently there would be no way
of forcing an update a la module update in pcbnew - unless I force a
delete of the cache).

In the end I'd still emphasize that believe it would be better, when
schematic designs (not only board files) would be "portable" without
pains, i.e. without dependencies on libs and without reminding of a
certain set of files that must be handed over. To recall Orcad (many years
ago...), I believe that the cache was inside the schematic design file.
And whis file WAS portable. eeschema's design file contains a list of used
lib files, but according to some experiments, this list is not used at all
(only when I also delete the very first LIBS: name in the .sch-file, it is
no longer recognized as eeschema file).

Let me excurse what happens when one omits the .pro file when he/she sends
a design:
- a default library list is created in the design. The design could be
rendered with errors
When anybody sends a .pro file (hypothetically) with deleted libs list:
- the design is rendered fully correct, because the cache is 100%
utilized, also no missing libs warnings.

This should help illustrate what can go wrong when you hand over a
schematic design for review. Certainly you know Murphy...

(wish myself goodnight now)

-Rob

On 12/6/2011 6:30 PM, Rob Maris wrote:
> Question #181052 on KiCad changed:
> https://answers.launchpad.net/kicad/+question/181052
>
> Status: Answered => Open
>
> Rob Maris is still having a problem:
>>> - I make a change in a component from the standard libs, *after* it has
>>> been inserted in the schematic before in a separate session. Hence, the
>>> cache holds the old version, the libs the altered version.
>>> - when this component change is saved, you can see it immediately in the
>>> schematic (i.e. behaviour a la relational database). Yet it is not
>>> written to the cache.
>>
>> The cache write doesn't happen until you save the schematic. It is not
>> updated automatically if a library component is changed. So if you
>> don't save the schematic, the cache will not be updated.
>
> Yes, I'm aware of that. I'm only describing what I'm obeying. In this case
> that a change in a library is immediately reflected in an open design
> (which is undesired - to my opinion).

I agree with your assessment but until the new schematic and library file
formats are implemented this known behavior will continue. The new library
concept will support multiple versions of the same part so the behavior you've
described will not happen not matter how many times you modify a part. This is
a long term project so you should not expect to see this change any time soon.
 In the mean time, do not modify existing parts. Make copies of them and give
them unique names so that existing parts in your schematic do not get changed.

>
>>> - then I add xxx-cache.lib on top of the lib preferences dialog
>>> "Component library files" and the new version is still visible.
>>
>> Because you really didn't add it to the beginning of the search list
>> (see code
>> above) it only appeared that way in the library search order dialog so
>> it was
>
> Yes, there is some asymmetry in the system's 'awareness' of changes: lib
> changes immediately; changes of search order after schematic reopen (the
> latter I expected).
>
>> still using the part in your custom library. The library dialog probably
>> should remove the cache from the sort order list to prevent confusion.
>> It is possible there is a bug where the dialog changes the sort order
>> without
>> calling the code above causing the sort order to be invalid until the
>> next time
>> Eeschema is restarted. I will take a look at it when I get a chance
>> because
>> this shouldn't happen. If it is happening I will file a bug report and
>> fix it.
>>
>>> - I exit and restart eeschema: the cache version is visible.
>>> - when I remove xxx-cache.lib from the list, after hitting the OK
>>> button,
>>> the changed version is visible immediately.
>>
>> This is expected if you didn't save the schematic to regenerate the
>> cache. You
>> probably shouldn't be able to delete the cache library either. It will
>> be
>
> Note that I do not delete the cache library, I removed its entry in the
> prirority list and for the test I intentionally wanted to exploit what is
> happening when there is a difference.
>
>> regenerated when you save your schematic and automatically added to the
>> end of the list the next time you open the schematic.
>
> Indeed, I did not save the schematic. When I would do that before exiting,
> then of course there would be no difference when I remove xxx-cache.lib
> from the priority list, because lib and cache are identical (regarding
> that sample component). However: what I believe to have proved is: the
> cache has priority over standard libs when I intentionally put the cache
> file on top of the list. So to my opinion, this priority is not
> 'automatically' destroyed by the algorithm (but perhaps there is some
> commutal misunderstanding...). Consider the following: I improve a
> schematic symbol in the standard libs, while NO schematic project is open
> (at least of project xyz). When I later open xyz, I'd find updated
> schematic symbols. However, when the cache would be entered into the list
> (on top), this would not happen (I know, currently there would be no way
> of forcing an update a la module update in pcbnew - unless I force a
> delete of the cache).
>
> In the end I'd still emphasize that believe it would be better, when
> schematic designs (not only board files) would be "portable" without
> pains, i.e. without dependencies on libs and without reminding of a
> certain set of files that must be handed over. To recall Orcad (many years
> ago...), I believe that the cache was inside the schematic design file.
> And whis file WAS portable. eeschema's design file contains a list of used
> lib files, but according to some experiments, this list is not used at all
> (only when I also delete the very first LIBS: name in the .sch-file, it is
> no longer recognized as eeschema file).
>
> Let me excurse what happens when one omits the .pro file when he/she sends
> a design:
> - a default library list is created in the design. The design could be
> rendered with errors
> When anybody sends a .pro file (hypothetically) with deleted libs list:
> - the design is rendered fully correct, because the cache is 100%
> utilized, also no missing libs warnings.
>
> This should help illustrate what can go wrong when you hand over a
> schematic design for review. Certainly you know Murphy...
>
> (wish myself goodnight now)
>
> -Rob
>

The entire EESCHEMA path search priority is poorly implemented. It has been the topic of several bug reports.
The core of the problem is that there is no reference to the originating library within the schematic, and the schematic relies on too much external data for its overall integrity.

I have addressed this conceptual problem in a fundamental re-design, in the form of the SWEET part fetching mechanism, which introduces the notion of "logical library name". The logical library name is present with each partname in the new design, side by side, sort of like: LogicalLibName.PartName. The logical library name is an index into a library table, where the full library path may reside.. The library table can be fully defined a) within the schematic, or b) portions of it can come from external places. a) lets you overcome your current problem.

This confusion that you are currently experiencing Rob, will go away with the new design change. I don't think trying to fix the current design is worth further discussion. It is like trying to fix a screen door on a submarine. The entire concept is a bug, so worrying about its location, its position, or is hinges are exercises in futility.