summaryrefslogtreecommitdiff
path: root/mirrors
AgeCommit message (Collapse)Author
2012-07-08mirrorcheck: Don't use bulk_create on sqlite3Dan McGee
It isn't worth it, as we run into the 999 max SQL statement variables issue when using it on any significant amount of mirrors. Since this is just a development database setup, and it isn't a command we need to run especially fast, we can ditch it. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-08Don't include StdDev on sqlite3 mirror status queryDan McGee
Because this function isn't shipped by default, it makes more sense to just omit it completely from the query we do to build the tables on this page when in development. Substitute 0.0 for the value so the rest of the calculations and display work as expected. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-08Handle HTTPException being thrown in mirrorcheckDan McGee
Managed to see this bubble up today when running the mirrorcheck command on a less than ideal connection that was experiencing timeouts at the wrong time. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-18mirrors: add an alternate_email columnDan McGee
We have a lot of these in the freeform text area in the mirror notes; attempt to make this data usable as necessary if we want to do some sort of mirror notification automation in the future. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-13Add ability to restrict status report to single tierrelease_2012-05-13Dan McGee
This should make it easier to catch errors in our Tier 1 mirrors. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-13Add rsync support to mirrorcheck and other small improvementsDan McGee
The main changes in this patch implement rsync:// protocol checking support by calling the rsync binary, requested in FS#29878. We track and log much of the same things as we already do for FTP and HTTP URLs- check time, last sync, total check duration, etc. Also added in this patch is a configurable timeout value which defaults to the previous hardcoded value of 10 seconds; this can be passed as an option to the mirrorcheck command. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-13Change mirror log error text to unlimited lengthDan McGee
Use TextField rather than a limited-length CharField; leave it up to the code filling this field to determine an appropriate length. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-13Don't limit protocols returned by mirror status functionDan McGee
If results weren't available for certain URLs, they won't show up anyway in this list, and if we start to check rsync URLs, then we want their values to come back in this status list. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-12Use python json module directly in place of simplejsonDan McGee
As of Python 2.6, this is a builtin module that has all the same functions and capabilities of the Django simplejson module. Additionally simplejson is deprecated in the upcoming Django 1.5 release. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-02Add a smart protocol filter to mirrorlist generatorDan McGee
This will only list FTP mirrors for a given country if there are no HTTP mirrors available, since FTP must die. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-02Allow mirrorlist generator pattern to match any protocolDan McGee
Add a helper method that checks if we know about the protocol; if so, we can spit out a URL for it. This allows (if you are insane) generation of an rsync mirrorlist, for instance. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-29Use a custom User-Agent when checking mirror URLsDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-28mirrorresolv: only run update query if values changedDan McGee
98% of the time, we won't need to update the existing values as it will be the same as the prior run of this command. Do a quick check of the old and new values and don't send anything to the database if there is no need for an update. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-26Remove now-unused NullCharFieldDan McGee
It would be straightforward if we just needed to remove the field class definition, but we also need to update all migrations that referenced it so we don't have problems starting and migrating from scratch. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-25Ensure sorted order of mirrors in status page matches with JSrelease_2012-04-25Dan McGee
We had one sorting order in the backend, and another once the JS sorting routine kicked in. Match them so we aren't doing more on the client-side on initial display than we have to. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-25Clean up mirrorlist generation form a bitDan McGee
Rename template since it really isn't an index of mirrors at all, and convert the form to use multiple checkboxes for both ftp/http and ipv4/ipv6 selection. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-25Finish django countries implementationDan McGee
* Add a migration to drop the old countries field. * Update all templates/views/utility methods to point at the new country field and dereference it as necessary. * Add the flags images to a few views where it makes sense. * Cleanup the download page layout quite a bit. * Bump the mirror status JSON version to 3; add country_code attribute. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-25Add django_countries country code fields and population migrationsDan McGee
This adds these columns and attempts to populate them with data from our existing country column data. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-25Rename mirror country fields to country_old in prep for normalizationDan McGee
We're going to move to using ISO 2 character codes via django countries, so start by moving the old data out of the way first. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-20Fix ContentType related migrationsDan McGee
This should fix the issues reported in FS#23228 for the most part. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-08Don't blow up when no mirror status data is availableDan McGee
The check here was wrong before; in the case of no mirror log entries the returned value will not be empty, but will contain two empty values. Check the values instead to see if we have valid data available. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-08Add migrations for on_delete attributesDan McGee
I don't think these end up doing anything at the database level, but since South insists on generating alter_column() calls for these changes, do them once so they don't keep reappearing in auto-generated future migrations. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-07Choose an up-to-date mirror for download URLsDan McGee
Given that we collect a lot of mirror status data, we can utilize it to ensure the download link on the website actually works and newly-added packages have actually been mirrored out. Add a method that attempts to use the mirror status data to determine a mirror we should redirect our download requests to. This can change on a regular basis, and falls back to the old method if no mirror status data is available. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-26Bump signoffs and mirror status JSON API versionsDan McGee
Due to datetime formatting changes in Django 1.4, we know follow the ECMA specification more closely and use 'yyyy-mm-ddThh:mm_ssZ' format. As this could break existing users of the JSON data, bump the version. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-24Use bulk_create() when inserting mirror log entriesrelease_2012-03-25Dan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-23Make all datetime objects fully timezone awareDan McGee
This is most of the transition to Django 1.4 `USE_TZ = True`. We need to ensure we don't mix aware and non-aware datetime objects when dealing with datetimes in the code. Add a utc_now() helper method that we can use most places, and ensure there is always a timezone attached when necessary. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-23Change Django urls.py importDan McGee
Until Django 1.3, the functions include(), patterns() and url() plus handler404, handler500 were located in a django.conf.urls.defaults module. In Django 1.4, they live in django.conf.urls. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-02-13Don't select FTP by default in mirrorlistsDan McGee
More generally, add a new 'default' column to the mirror protocol model so we can determine what is selected and shown by default. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-01-05Adjust page and content caching lengths and decoratorsDan McGee
Remove never_cache from many places now that we don't actually need it since we aren't caching by default. Adjust our cache_function decorator times be shorter values, and also randomize them a bit to make cache invalidations not all line up. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-12-11PyLint suggested cleanupsDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-12-11Switch back to using standard deviation in mirror check pageDan McGee
This got checked in by default, whoops. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-12-07Various small admin touchupsDan McGee
* Don't capitalize things in verbose_name, Django does this automatically and uses title case * Add overrides for IP, URL, ISO, etc. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-10Add package signoffs JSON viewDan McGee
This allows access to the same data (and even a bit more) from the signoffs overview page in a machine-friendly way. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-10-26Don't require mirrorlist URLs to have ISOsrelease_2011-10-26Dan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-10-11Pylint suggested and other cleanupsDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-08-22Admin setup tweaksDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-08-16Small Python convention fixupDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-07-29mirrors: do protocol formatting in templateDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-14Add JSON package details and files viewsDan McGee
These are retrieved by adding 'json/' to the normal package details or files view. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-14Allow mirror country field to be persisted to DB as NULLDan McGee
You need a custom field type in Django to allow this. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-09Management command cleanupDan McGee
Now that we aren't seeing odd segfaults and hung tasks, we can remove the traceback stuff from the scripts. Also use the 'io' module only, it has been long enough. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-18mirrors: pylint discovered cleanupsDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-18Never parse generated lastsync fileDan McGee
We should be encouraging our mirrors to serve us the original file, not something they create and come up with. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-12Add optional country override for individual mirror URLsDan McGee
This allows a named top-level mirror to have geographically distributed URLs, e.g. kernel.org and the geo-DNS setup. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-07Utilize Django 1.3 'on_delete' feature on several foreign keysDan McGee
The most important one here is PROTECT to keep people from making bone-headed plays and deleting an Arch or Repo and every package along with it. We can use this in a few other places, as well as some carefully placed SET_NULL indicators. Note that nothing here pushes deletion responsibilities down to the database, although that will probably happen in a future commit. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-05Make mirror name and URLs uniqueDan McGee
So we don't create duplicates without knowing it in our database, like we did earlier today. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-28Enhance the mirror details pageDan McGee
Get the URLs with their performance data showing up, and simplify the top part for non-authenticated users while adding more detail for logged-in users. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-28Move country filter last in mirror adminDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-03Add ordering to mirror protocolsDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-02-27Auto-resolve mirror URLs on saveDan McGee
This prevents people from having to mess with these checkboxes at all in the admin, and we incur no delay on their initial values being correct waiting for the cron job to run. Signed-off-by: Dan McGee <dan@archlinux.org>