summaryrefslogtreecommitdiff
path: root/devel/management/commands/reporead.py
AgeCommit message (Collapse)Author
2012-08-08Extract parse_version function from reporead logicDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-08-04reporead: import make and check dependsDan McGee
We don't have these in the database yet, but future verisons of repo-add will put this information in the sync databases. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-08-04Make adjustments for optional -> deptype conversionDan McGee
Very little dealt directly with this field. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-28reporead: don't use iexact lookup on arch nameDan McGee
We don't do this anywhere else, so we shouldn't do this here either. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-24Remove custom utc_now() function, use django.utils.timezone.now()Dan McGee
This was around from the time when we handled timezones sanely and Django did not; now that we are on 1.4 we no longer need our own code to handle this. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-08Work around bulk_create limitations in sqlite3 in reporeadDan McGee
Given the 999 SQL statement variable limit, we can easily hit it when updating a package with thousands of files or a few hundred depends. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-08reporead: disable FULL synchronous writes for sqlite3Dan McGee
At least on Linux, we hit a huge bottleneck waiting for the FULL commit to happen for each added package during reporead operations. It makes much more sense to back this off to FULL level instead, which trades some possible loss of durability for speedier operation. Additionally, no one would possibly be running their production version of this site on sqlite3, right? Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-05reporead: don't append slash to empty (root) directoryDan McGee
Add the slash only if we have a directory name, and not otherwise. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-05reporead: handle files in root directory properlyDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-05reporead: properly handle cases where last_update == files_last_updateDan McGee
We should assume the filelists are up to date in this case, not out of date. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-01Log package updates during reporead invocationDan McGee
This adds a Manager and log_update method to help log all updates made to the packages table during reporead runs. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-19reporead: fix copy/paste issueDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-18Switch to usage of new Depend objectDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-26Rename 'packagedepend_set' attribute to 'depends'Dan McGee
We do this for every other related package attribute, so do it here too. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-23reporead: use bulk_create() for more propertiesDan McGee
Depends, conflicts, provides, etc. can all be done via bulk_create. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-23Merge branch 'django14'Dan McGee
Conflicts: templates/releng/result_section.html
2012-03-23Make all datetime objects fully timezone awareDan McGee
This is most of the transition to Django 1.4 `USE_TZ = True`. We need to ensure we don't mix aware and non-aware datetime objects when dealing with datetimes in the code. Add a utc_now() helper method that we can use most places, and ensure there is always a timezone attached when necessary. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-23reporead: use Django 1.4 bulk_create() for package filesDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-23reporead: use Django 1.4 select_for_update()Dan McGee
As per TODO comments in the existing code. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-23reporead: blow up when package found with wrong architectureDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-16reporead: rename Pkg to RepoPackageDan McGee
The bytes saved on the shorter name aren't worth it. Also ensure 'desc' is always initialized to None in case packages do not provide one. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-02-11reporead: only reset flag date if upstream version changesDan McGee
This preserves the flag date if only a simple pkgrel bump occurred, which makes sense more often than not for rebuilds. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-01-19reporead: simplify and fix transaction management in update_common()Dan McGee
We can use the easier transaction.commit_on_success() decorator if we be sure to explicitly mark the transaction dirty. This fixes the issue where a raised exception in this code called neither commit nor rollback. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-12-12reporead: more efficient deletion of filesDan McGee
Rather than delegating to Django and batch deletion by ID, force issuing of a single delete query to clear out all existing file objects when necessary. This should speed up the deletion and update of packages with a lot of files by a non-trivial amount. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-12-11PyLint suggested cleanupsDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-12-03reporead: don't update timestamp on --forceDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-12-03reporead: fix --force flagDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-30reporead: fix not defined variableDan McGee
Way to fail at refactoring, Dan. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-30reporead: split out filesonly update methodDan McGee
This removes a bunch of the conditional logic at a slight cost of some code duplication. However, the methods and madness is now much easier to follow. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-30reporead: fix filesonly needs update checksDan McGee
This was broken after the select for update changes. We really should split the whole filesonly update into another method instead of the current shotgun approach with conditionals everywhere. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-17reporead: don't trim pkgdesc lengthDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-17Ensure reporead is protected against simultaneous runsDan McGee
This adds a bunch of transaction magic and SELECT FOR UPDATE stuff to reporead to cope with the now-concurrent runs of reporead we get when invoked from our inotify-based updater. The collision occurs with 'any' architecture packages as both repo databases contain the new version, and the updates occur at exactly the same time. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-16reporead: a few small tweaksDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-15reporead: clean up some debug loggingDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-15Improve primary arch validationDan McGee
Ensure we can accept either a Arch object or an architecture name when passed to read_repo() by moving the validation there and being a bit more careful about typechecking and object lookup. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-10-26Ensure PGP signature values are not trimmedDan McGee
This makes them totally unusable for any real purpose down the road. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-08-17Add PGP signature package fieldDan McGee
And add eventual display code for it to the details template, but don't show it yet as no packages will have it. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-08-16Add two new DB fields to reporeadDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-23Turn find_user into UserFinder classDan McGee
This moves the cache inside an instance. Also add a few more tests. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-23Move find_user method to devel utilsDan McGee
This could be handy elsewhere as well, and it is loosely coupled to anything else in reporead. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-14reporead: small memory/perf improvementsDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-14reporead: two small cleanupsDan McGee
* Parse builddate when reading from repo database file * Use defaultdict where it comes in handy Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-09Fix busted batch score on package removalDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-09reporead: allow batching of package updatesDan McGee
The real reason I originally added transactions to this code was to prevent half-updates; e.g. a package gets in without the matching depends values. We can safely commit between packages and resume processing the database at a later time. Take advantage of this fact and commit every so often in batch fashion if we have a lot of updates piling up. In the case of updating the files DB, this can really cut down on the need to hold open a long-running, statement heavy transaction and get the information public faster. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-09Management command cleanupDan McGee
Now that we aren't seeing odd segfaults and hung tasks, we can remove the traceback stuff from the scripts. Also use the 'io' module only, it has been long enough. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-07Use UTC datetime objects everywhereDan McGee
Rather than the twisted mix of local times and UTC times we currently have. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-24Add provides to collections listrelease_2011-03-28Dan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-23reporead: remove the need for hasattr() checksDan McGee
Ensure all our multivalued attributes already exist on the object beforehand, and add some special sauce to handle the difference between a package without files and a database without files entries. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-23Clear out package relation sets before adding new valuesDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-23reporead: read in provisions, conflicts, and replacementsDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
generated by cgit v1.2.3-2-g168b (git 2.25.1) at 2024-07-08 19:49:30 +0000