file/folder search in the backups

Hi Kodanda!

Great! Code contributions are always very welcome! :)

I've no plans on implementing this at the moment. But if you would like to add this feature I'm happy to help you on this.

I do have plans on implementing some kind of background cache crawler for QFileSystemModel. This should visit and cache all folders in current folder and the parent folder and also the current folder in all other snapshots. So if you switch folders or snapshot the new folder will show up immediately. Specially with mode SSH or SSH-encrypted it could take couple seconds at the moment which makes it hard to browse in snapshots.

But I wasn't successful on this till now.

So, with this said I think it would be best to implement a search feature also based on QFileSystemModel instead of bare filesystem.

Kind regards,
Germar

PS: feel free to contact me on Jabber if you like

Hi Germar,

Thanks you for the reply. As I have mentioned before, I have already started working on this idea and got a fair amount of it to work. See this file for some snapshots

http://www.filedropper.com/bitfilesearchrestore

I am not experienced in python so much, so I wrote it in Qt/C++. It is based on QFileSystemModel+TreeSortFilter for filtering.

I was trying find matching files/folders given an RegEx, and present all matching items from one snapshot to the user. This much has been implemented by now and seems to work pretty well.

For file restoration I send calls to backintime CLI from within my app. I have tested it on local back-ups only.

Is is fine (strategy wise) to have both C++ and Python parts in BIT? May be we can call this app from within the main backintime-qt code. Suggest me how to go further.

Best regards,
Kodanda

PS: this is my first-ever contribution to FOSS.

Sorry, I was busy the last days but I'll look into this soon!

Congrats and Thanks for your first FOSS contribution!!! :D

Hi Gemar,

Thanks for the reply. I have browsed through the python/qt code from the trunk. Though I have little experience in python, it didn't look that scary, I could manage to translate the C++/Qt part into python with some effort.

Given that, I now feel that it is better to implement the same search feature into the pyQt. My idea would be to work on the files_view only, such that
(1) remove the file path (QLineEdit) and Up arrow button
(2) change the file_list_view into fully tree-like view with the snapshot folder always begin the root folder. (just like in my app, see the screenshots in the previous email)
(3) leave other Hidden file, restore, etc. buttons as they are.

What do you think?

Cheers,
Kodanda

Hi Kodanda,

That sounds great! The screenshots already looked promising but it would be lot more intuitive to use this directly inside BIT.

I think it would be best to have the search inside an own QThread so it wouldn't freeze the main-window thread and could crawl all snapshots in background. Maybe we could highlight those snapshots (in list_time_line) with matches one after an other.

I remember I was thinking about changing list_files_view into this tree-like view (QTreeView.setRootIsDecorated(True) and QTreeView.setItemsExpandable(True)) when I was porting BIT to pure Qt4. But there was some drawback I can't remember at the moment that let me revoke this. Only problem I could imagine at the moment would be to expand the same structure again after switching snapshots.
We would need to adjust all functions which use selections from list_files_view. But that shouldn't be a big issue.

For performance there should be controls about where to search (whole filesystem recursive | current folder recursive | only current folder)

Kind regards,
Germar

Hi Gemar,

I also meant to fuse the search feature in the native GUI of BIT. About the GUI modifications, see the following snapshot:

http://s725.photobucket.com/user/mkodanda/media/BIT%20screenshots/201503250939_zpsbf6hhklm.png.html

The Idea behind the whole tree view (of one single snapshot) is the following use case: The user vaguely remembers the name of the file, so he searches for files matching a keyword. He does not remember where he stored it originally. So the search in BIT will have to present a list of matching FILES & FOLDERS IN ONE SNAPSHOT, showing him the directory tree view.

Regarding the search controls for performance:
I use a subclassed version of the QSortFilterProxyModel (called TreeSortFilter) for searching; but I am not sure about the internal filtering mechanism, whether it applies the filter on the whole filesystem model OR starts to apply the filter from the rootIndex in the treeView display is not clear to me. It looks like it filters the whole model, but may be depending on the view the directory loading sequence could differ, and thus the search results.

Cheers,
Kodanda

PS:
In future, my idea is to make a dolphin service action menu: User right-clicks on a file/folder in the original file system, and selects "Restore" or "Restore To" from the service actions menu. In this case a window pops up and presents him all the snapshot IDs where an EXACT match of the selected entry exists in the backup.

Hi Gemar,

A License related question:

I have implemented the filter, and it seems to work, provided the directories are fully loaded. However, I translated into Python a part of the TreeSortFilter.cpp (first appeared as snippets on forums in 2006, and now comes in GPL & LGPL by Nokia (c) 2009). Is it legal to use this in BIT code?

Cheers,
Kodanda

Hi Gemar and others,

Please give a try using the code in my branch:

lp:~kodanda/backintime/incorp-bit-search

It now includes search functionality on the files and folders. All the matching entries are shown in a tree view for a clearer and easy identification by the user. Search is performed under the current directory selected (displayed in the address tool bar). Only after expanding the tree below, the file watcher populates the data for search. In future this crawling could be separated out into an independent thread.

The screen shots are here:

http://s725.photobucket.com/user/mkodanda/library/BIT%20screenshots

Let me know your feed back.

Cheers,
Kodanda

Hi Kodanda,

sorry for my lousy response time. Quite busy week.

It looks great! I just had a quick look and would like to go deeper into this (and answer all your questions) tomorrow. But I think you forgot to add the treesortfilter class. You need to run 'bzr add qt4/treesortfilter.py' and commit & push it.

BIT is licensed under GPLv2, too. So using other GPL source should be fine.

Kind regards,
Germar

Hi Germar,

Added the missing treesortfilter.py to the branch. Please see its header for the original license. It contains GPL, LGPL, and what not, all :)

But the snippets of filter code I used can also be found in forums before 2009.

Best Regards,
Kodanda

Hi Kodanda,

I played a bit with your branch and would like to share some thoughts in random order (just as they came to my mind):

1. I like the way you solved the tree-like view. This makes it lot easier to navigate.

2. the 'Expand Tree' button does only expand one level (Qt 4.8.5). I think it would be consequent to expand the whole tree at once.

3. we need a 'Collapse Tree' button, too.

4. I like the way the search-bar drops down in both Gedit[1] and Nautilus[2]. Maybe we could put the QLineEdit and all buttons that belong to search into an extra drop-down menubar, too? If QLineEdit is empty it should vanish after e.g. 10sec. Would clean up the toolbar.

5. Hitting return or pressing search should expand the tree (at least to all matches). I know why it doesn't at the moment because this is a filter not a search. But this would be very irritating for users who expect a search behind a search button ;-)

6. search pattern should stay after switching snapshots in timeline. This way your use case from answer #6 could be expanded to browse in different snapshots for the missing file.

6a. In future when we have this search feature running in a separate thread we could search the other snapshots in background and highlight all snapshots with matches in timeline.

7. would be great if the expandation of the tree would stay the same after switching snapshots.

8. folders should expand only when pressing the arrow in front. Double click should still open the folder.

9. could you please post the original source of TreeSortFilter? IMHO the invalidateFilter method is obsolete as it does the same as calling the subclassed method directly.

10. please don't use wildcard import for treesortfilter. Instead you could paste TreeSortFilter into qt4tools.py and call it explicit with qt4tools.TreeSortFilter()

Kind Regards,
Germar

[1]: https://architekt-reitze.de/owncloud/public.php?service=files&t=4462e8f8b19b11fb0afc7b7370d1b8e7
[2]: https://architekt-reitze.de/owncloud/public.php?service=files&t=81c0312fa73f0defbbad190152d79392

Hi Germar,

A collapse button is now added.

Your points 5, 6, 6a, and 7 are all actually waiting for the search feature to have its own thread :) Once the fileModel is fully populated these things can be achieved without freezing the GUI.

The expand button (populating the tree below), and thus manually selecting the depth of search/filtering is related to freezing of the GUI to expandAll items in the tree. Once, I get the search in a separate thread, there would be no need for the expandAll and collapseAll buttons.

9. The invalidateFilter() is listed as a private member in the documentation (Qt 5.4). Is it getting deprecated in the next Qt versions? Or you mean something else? BTW, this invalidateFilter() methods wasn't there in the original TreeSortFilter. I added it to give access to the parent class' private method.

Best,
Kodanda

PS: Here is the TreeSortFilter.cpp:
/****************************************************************************
**
** Copyright (C) 2009 Nokia Corporation and/or its subsidiary(-ies).
** Contact: Nokia Corporation (<email address hidden>)
**
** This file is part of the Patternist project on Trolltech Labs.
**
** $QT_BEGIN_LICENSE:LGPL$
** Commercial Usage
** Licensees holding valid Qt Commercial licenses may use this file in
** accordance with the Qt Commercial License Agreement provided with the
** Software or, alternatively, in accordance with the terms contained in
** a written agreement between you and Nokia.
**
** GNU Lesser General Public License Usage
** Alternatively, this file may be used under the terms of the GNU Lesser
** General Public License version 2.1 as published by the Free Software
** Foundation and appearing in the file LICENSE.LGPL included in the
** packaging of this file. Please review the following information to
** ensure the GNU Lesser General Public License version 2.1 requirements
** will be met: http://www.gnu.org/licenses/old-licenses/lgpl-2.1.html.
**
** In addition, as a special exception, Nokia gives you certain
** additional rights. These rights are described in the Nokia Qt LGPL
** Exception version 1.0, included in the file LGPL_EXCEPTION.txt in this
** package.
**
** GNU General Public License Usage
** Alternatively, this file may be used under the terms of the GNU
** General Public License version 3.0 as published by the Free Software
** Foundation and appearing in the file LICENSE.GPL included in the
** packaging of this file. Please review the following information to
** ensure the GNU General Public License version 3.0 requirements will be
** met: http://www.gnu.org/copyleft/gpl.html.
**
** If you are unsure which license is appropriate for your use, please
** contact the sales department at http://www.qtsoftware.com/contact.
** $QT_END_LICENSE$
**
***************************************************************************
*/

#include <QtDebug>

#include "TreeSortFilter.h"

using namespace QPatternistSDK;

TreeSortFilter::TreeSortFilter(QObject *p) : QSortFilterProxyModel(p)
{
    Q_ASSERT(p);
}

bool TreeSortFilter::lessThan(const QModelIndex &left,
                              const QModelIndex &right) const
{
    const QVariant leftData(sourceModel()->data(left));
    const QVariant rightData(sourceModel()->data(right));

    return numericLessThan(leftData.toString(), rightData.toString());
}

bool TreeSortFilter::numericLessThan(const QString &l, const QString &r) const
{
    QString ls(l);
    QString rs(r);
    const int len = (l.length() > r.length() ? r.length() : l.length());

    for(int i = 0;i < len; ++i)
    {
        const QChar li(l.at(i));
        const QChar ri(r.at(i));

        if(li >= QLatin1Char('0') &&
           li <= QLatin1Char('9') &&
           ri >= QLatin1Char('0') &&
           ri <= QLatin1Char('9'))
        {
            ls = l.mid(i);
            rs = r.mid(i);
            break;
        }
        else if(li != ri)
            break;
    }

    const int ld = ls.toInt();
    const int rd = rs.toInt();

    if(ld == rd)
        return ls.localeAwareCompare(rs) < 0;
    else
        return ld < rd;
}

bool TreeSortFilter::filterAcceptsRow(int sourceRow, const QModelIndex &sourceParent) const
{
    if(filterRegExp().isEmpty())
        return true;

    QModelIndex current(sourceModel()->index(sourceRow, filterKeyColumn(), sourceParent));

    if(sourceModel()->hasChildren(current))
    {
        bool atLeastOneValidChild = false;
        int i = 0;
        while(!atLeastOneValidChild)
        {
            const QModelIndex child(current.child(i, current.column()));
            if(!child.isValid())
                // No valid child
                break;

            atLeastOneValidChild = filterAcceptsRow(i, current);
            i++;
        }
 // New: added to accept if the current memeber contains the filter expression
     bool currentAccepted=sourceModel()->data(current).toString().contains(filterRegExp());
        return atLeastOneValidChild || currentAccepted;
    }

    return sourceModel()->data(current).toString().contains(filterRegExp());
}

// New
// allow to access invalidateFilter()
void TreeSortFilter::invalidateFilter()
{
    QSortFilterProxyModel::invalidate();
}

// vim: et:ts=4:sw=4:sts=4

Hi Kodanda,

to be honest I've never understood the concept of private/protected members in C++ as Python doesn't have such. But calling QSortFilterProxyModel().invalidateFilter() from python works fine and so it should work in a subclass, too.

BTW preferred way to call subclassed __init__ in Python is to run 'super(TreeSortFilter, self).__init__()' (in line 48)

Kind regards,
Germar

This question was expired because it remained in the 'Open' state without activity for the last 15 days.