Archive for the ‘Moodle’ Category

Finding duplicate entries in MySQL database

Thursday, July 7th, 2011

On upgrading my Moodle installation from version 1.9 to 2.1 the upgrade process (initiated from command line) exited with the following error:

!!! Error reading from database !!!
!! Illegal mix of collations (utf8_general_ci,IMPLICIT) and (utf8_unicode_ci,IMPLICIT) for operation '='
SELECT po.id AS oldpage_id, po.pagename AS oldpage_pagename, po.version, po.flags,
                   po.content, po.author, po.userid AS oldpage_userid, po.created, po.lastmodified, po.refs, po.meta, po.hits, po.wiki,
                   p.id AS newpage_id, p.subwikiid, p.title, p.cachedcontent, p.timecreated, p.timemodified AS newpage_timemodified,
                   p.timerendered, p.userid AS newpage_userid, p.pageviews, p.readonly, e.id AS entry_id, e.wikiid, e.course AS entrycourse,
                   e.groupid, e.userid AS entry_userid, e.pagename AS entry_pagename, e.timemodified AS entry_timemodified,
                   w.id AS wiki_id, w.course AS wiki_course, w.name, w.summary AS summary, w.pagename AS wiki_pagename, w.wtype,
                   w.ewikiprinttitle, w.htmlmode, w.ewikiacceptbinary, w.disablecamelcase, w.setpageflags, w.strippages, w.removepages,
                   w.revertchanges, w.initialcontent, w.timemodified AS wiki_timemodified,
                   cm.id AS cmid
              FROM wiki_pages_old po
              LEFT OUTER JOIN wiki_entries_old e ON e.id = po.wiki
              LEFT OUTER JOIN wiki w ON w.id = e.wikiid
              LEFT OUTER JOIN wiki_subwikis s ON e.groupid = s.groupid AND e.wikiid = s.wikiid AND e.userid = s.userid
              LEFT OUTER JOIN wiki_pages p ON po.pagename = p.title AND p.subwikiid = s.id
              JOIN modules m ON m.name = 'wiki'
              JOIN course_modules cm ON (cm.module = m.id AND cm.instance = w.id)

Summed up — an “Illegal mix of collations” was occurring while querying the table wiki_pages_old. A quick search found a possible solution. Unfortunately (for now) I’m not really sure if it helped, since the error persisted, but it is worth of making a note of it until being sure (by migrating the production server).

So I went in with phpMyAdmin and changed the collation on the table manually, which didn’t help either, since it didn’t change the collation of the individual fields. So I changed the fields’ collation manually, except for one field, which — after changing the collation — started to report a duplicate entry. After some trial and error I’ve found a solution.

From the table dump file I’ve deleted the offending key and after importing the table into the database I’ve searched for the offending duplicate entries.

The following MySQL query, from MySQL forum, finds the duplicate entries.

SELECT t1.* FROM t1 INNER JOIN (
SELECT colA,colB,COUNT(*) FROM t1 GROUP BY colA,colB HAVING COUNT(*)>1) as t2
ON t1.cola = t2.cola and t1.colb = t2.colb;

In my case the upper query changes into this:

SELECT wiki_pages_old.* FROM wiki_pages_old INNER JOIN (
SELECT pagename,version,wiki,COUNT(*) FROM wiki_pages_old GROUP BY pagename,version,wiki HAVING COUNT(*)>1) as wiki_t2
ON wiki_pages_old.pagename = wiki_t2.pagename and wiki_pages_old.version = wiki_t2.version and wiki_pages_old.wiki = wiki_t2.wiki;

I’ll write some more details after migrating the production server.

Installing Alternative PHP Cache (APC) on Debian Etch

Wednesday, January 9th, 2008

Obtaining and installing APC

Alternative PHP Cache (APC) is a free, open, and robust framework for caching and optimizing PHP intermediate code. It’s an PECL extension which shares the packaging and distribution system with its sister, PEAR.

Provided that you have PEAR (php-pear) package installed on your system, the procedure is as simple as pecl install apc. Alternatives.

meglohvat:# pecl install apc
downloading APC-3.0.16.tgz ...
Starting to download APC-3.0.16.tgz (114,298 bytes)
.............done: 114,298 bytes
45 source files, building
running: phpize
Configuring for:
PHP Api Version:         20041225
Zend Module Api No:      20060613
Zend Extension Api No:   220060519
Use apxs to set compile flags (if using APC with Apache)? [yes] :

As probably corectly guessed by PECL package managers, you are runing an Apache web server so you should confirm by typing yes or hitting Enter key.

Errors which you may (I did) encounter

checking for re2c... no
configure: WARNING: You will need re2c 0.9.11 or later if you want to \
regenerate PHP parsers.

The re2c package was missing on my system ›› apt-get install re2c.

checking for gawk... no
checking for nawk... nawk
checking if nawk is broken... no

I had no gawk or nawk but mawk pattern scanning and processing language installed on system ›› apt-get install gawk.

checking whether apc needs to get compiler flags from apxs...

Sorry, I was not able to successfully run APXS.  Possible reasons:

1.  Perl is not installed;
2.  Apache was not compiled with DSO support (--enable-module=so);
3.  'apxs' is not in your path.  Try to use --with-apxs=/path/to/apxs
The output of apxs follows
/tmp/tmpJQuZdD/APC-3.0.16/configure: line 3846: apxs: command not found
configure: error: Aborting
ERROR: `/tmp/tmpJQuZdD/APC-3.0.16/configure --with-apxs' failed

Missing APache eXtenSion (APXS) tool for building and installing extension modules for the Apache HyperText Transfer Protocol (HTTP) server. APXS is found in apache2-threaded-dev package. The command apt-get install apache2-threaded-dev will also install a long chain of dependencies.

Setting up APC

A good place to start is Alternative PHP Cache chapter in PHP manual. There it’s written that the default configuration is sutable for most instalations, but they single out two settings: apc.shm_size and apc.stat.

apc.shm_size
Here you should enter the size of each shared memory segment in MB, the default value being 30MB. By default, the maximum size of shared memory segment in Debian is 33554432 or 33MB. You can check it by cat /proc/sys/kernel/shmmax command. If you want to know more about shared memory segments, read this forum thread.
UPDATE: “Of course you can always raise the maximum size of a shared memory segment, using sysctl. Do this only when you know what you are doing. Run “man sysctl” for more information.” (source).
UPDATE: I’ve increased the apc.shm_size to 256MB without changing any kernel setting and it works fine.
apc.stat
From PHP manual:
Be careful if you change this setting. The default is for this to be On which means that APC will stat (check) the script on each request to see if it has been modified. If it has been modified it will recompile and cache the new version. If you turn this setting off, it will not check. That means that in order to have changes become active you need to restart your web server. On a production server where you rarely change the code, turning stats off can produce a significant performance boost.

For included/required files this option applies as well, but note that if you are using relative path includes (any path that doesn’t start with / on Unix) APC has to check in order to uniquely identify the file. If you use absolute path includes APC can skip the stat and use that absolute path as the unique identifier for the file.

If you change these settings or not, the next step is to move the /usr/share/php/apc.php script into your webserver path and access it with browser. It provides a detailed look at what is happening with your cache and, with GD enabled in PHP, also shows you graphs of the situation of your cache.

After checking if caching actualy works, you should point your eyes to the Cache full count value (on the left tables under File Cache Information). The number tells you how many times the cache filled up the allocated memory and had to be cleared of entries not accessed within number of seconds set up with apc.ttl setting. “You should configure your cache to minimize this number if not the resulting cache churn is going to hurt performance. You should either set more memory aside for APC, or use apc.filters to cache fewer scripts.”*

Updates

When I tried to replicate the above installation process on my production server running on minimum required software I encountered some additional problems/missing packages.

    meglohvat:# pecl install apc
    downloading APC-3.0.16.tgz ...
    Starting to download APC-3.0.16.tgz (114,298 bytes)
    .........................done: 114,298 bytes
    45 source files, building
    running: phpize
    sh: phpize: command not found
    ERROR: `phpize' failed

phpize is a shell script to prepare PHP extension for compiling. It is a part of php5-dev package ›› apt-get install php5-dev

Upgrade

To upgrade the APC extension run the command:

meglohvat:# pecl upgrade apc
downloading APC-3.0.19.tgz ...
Starting to download APC-3.0.19.tgz (115,735 bytes)
.........................done: 115,735 bytes
47 source files, building
running: phpize
Configuring for:
PHP Api Version:         20041225
Zend Module Api No:      20060613
Zend Extension Api No:   220060519
Use apxs to set compile flags (if using APC with Apache)? [yes] :

And if everything went well you should see something like this in the end.

Build process completed successfully
Installing '/var/tmp/pear-build-root/install-APC-3.0.19//usr/lib/php5/20060613/apc.so'
upgrade ok: channel://pecl.php.net/APC-3.0.19
You should add "extension=apc.so" to php.ini
meglohvat:#

Moodle: session timeout problem

Thursday, January 18th, 2007

Even though I have set-up a 4 hours sessiontimeout period in Moodle settings, my users were thrown out of their sessinos in 24 minutes.

I searched the moodle.org web-page and found a soulution on this page: Using Moodle: CRON job and sessions relation.

In short: the problem was that PHP setting (php.ini) session.gc_maxlifetime was set-up to 1440 seconds (24 minutes). I changed this to 3600 seconds (1 hour) and now it works.