2008-05-11T00:07:25 <ThomasWaldmann> btw, some Twisted and mod_python stuff still needs testing, if anybody cares about this stuff :)
2008-05-11T00:10:12 <dreimark> I've run them several times without seeing this timing problem. pagesaving sends the event to get wikidictsrescan
2008-05-11T00:12:18 <dreimark> if there is in between these a timing problem then it can cause an update problem of the dict
2008-05-11T00:16:06 <ThomasWaldmann> you maybe need a faster machine or slower clock :)
2008-05-11T00:16:46 <dreimark> do you think it depends on the length of the page too.
2008-05-11T00:16:58 <dreimark> default is 3000 users and one test with 30000
2008-05-11T00:18:17 <ThomasWaldmann> i just ran py.test in the toplevel dir
2008-05-11T00:18:45 * dreimark tests finished: 635 passed, 1 failed, 64 skipped in 104.89 seconds
2008-05-11T00:18:56 <ThomasWaldmann> (my machine is rather slow 1GHz/1GB, but has a flash based disk)
2008-05-11T00:21:16 <ThomasWaldmann> tests finished: 617 passed, 2 failed, 67 skipped in 78.94 seconds
2008-05-11T00:21:55 <ThomasWaldmann> E return self.groupdict[name]
2008-05-11T00:21:55 <ThomasWaldmann> > KeyError: u'AnotherGroup'
2008-05-11T00:21:55 <ThomasWaldmann> [/home/tw/eclipse-ws/moin-1.7/src/MoinMoin/wikidicts.py:236]
2008-05-11T00:23:47 <dreimark> that is an older test. Is always this failing?
2008-05-11T00:23:59 <ThomasWaldmann> running it again: only 1 failure
2008-05-11T00:25:32 <ThomasWaldmann> E assert result == "This is a brand new example"
2008-05-11T00:25:32 <ThomasWaldmann> > assert u'This is an example' == 'This is a brand new example'
2008-05-11T00:25:44 <ThomasWaldmann> now 2 failures
2008-05-11T00:25:53 <ThomasWaldmann> [/home/tw/eclipse-ws/moin-1.7/src/MoinMoin/macro/_tests/test_GetVal.py:65]
2008-05-11T00:26:59 <dreimark> the first calls page.renampePage os.rename(oldpath, newpath)
2008-05-11T00:27:50 <dreimark> flash r/w is asynchronous so it can be slower in the renaming
2008-05-11T00:28:53 <ThomasWaldmann> no
2008-05-11T00:29:11 <ThomasWaldmann> now 1 failure total again
2008-05-11T00:34:33 <dreimark> why does my system think I do like to have only 1GHz
2008-05-11T00:38:14 <dreimark> ok now with 2.2GHz
2008-05-11T00:38:16 <dreimark> tests finished: 635 passed, 1 failed, 64 skipped in 54.97 seconds
2008-05-11T00:39:25 <dreimark> tests finished: 635 passed, 1 failed, 64 skipped in 55.30 seconds
2008-05-11T00:42:20 <ThomasWaldmann> do you have a ramdisk? tmpfs?
2008-05-11T00:43:58 <dreimark> currently not, can add one
2008-05-11T00:44:31 <dreimark> we can add for a test time.sleep(0.2) after page.renamePage('AnotherGroup')
2008-05-11T00:48:46 * ThomasWaldmann will do self.sleep() now :)
2008-05-11T00:48:53 <dreimark> good night
2008-05-11T00:48:59 <ThomasWaldmann> gn
2008-05-11T00:49:03 <dreimark> seen 65MB is the default size here
2008-05-11T01:09:32 <dreimark> ThomasWaldmann: confirmed
2008-05-11T01:27:57 <dreimark> gn
2008-05-11T09:37:54 <dreimark> mmoin
2008-05-11T09:37:58 <dreimark> I think we should not backport 1.6.3FullSearchWithXapianForCategory
2008-05-11T09:38:00 <dreimark> we can provide a patch for 1.6 on the page so only people who ran into this and wanted it fixed can get it
2008-05-11T09:46:38 * dreimark works on wikidictsrescan
2008-05-11T11:13:33 <CIA-53> Reimar Bauer <rb.proj AT googlemail DOT com> default * 3574:0b7eb697e952 1.7/MoinMoin/_tests/test_wikidicts.py: test_wikidicts: PEP8 whitespace fix
2008-05-11T11:15:17 <ThomasWaldmann> moin
2008-05-11T11:16:19 <ThomasWaldmann> dreimark: seems that the sourcecodechecker does not fail on these
2008-05-11T11:17:26 <dreimark> http://moinmo.in/ReimarBauer?action=AttachFile&do=view&target=wikidictsrescan_patch.patch
2008-05-11T11:17:43 <dreimark> ThomasWaldmann: yeah I was wondering too
2008-05-11T11:18:03 <dreimark> ThomasWaldmann: I have no failing dict test with that patch
2008-05-11T11:18:16 <dreimark> on my ramdisk too
2008-05-11T11:21:33 <ThomasWaldmann> ...because it can be interrupted?
2008-05-11T11:22:07 <dreimark> The actual suspension time may be less than that requested because any caught signal will terminate the sleep() following execution of that signal's catching routine.
2008-05-11T11:22:32 <dreimark> explaination from http://docs.python.org/lib/module-time.html
2008-05-11T11:23:10 <ThomasWaldmann> what signals do you expect?
2008-05-11T11:27:21 <ThomasWaldmann> and also please explain why and how this patch fixes the problem
2008-05-11T11:27:25 <dreimark> http://blog.nickvonhollen.com/2007/05/sleep-in-python-should-never-be-trusted.html
2008-05-11T11:29:39 <ThomasWaldmann> that doesnt answer my question
2008-05-11T11:30:03 <dreimark> I know I tried before with sleep currently it works too
2008-05-11T11:31:40 <ThomasWaldmann> the code in you patch might run around in circles for up to a second, consuming 100% cpu, that is not acceptable
2008-05-11T11:32:11 <dreimark> yeah, it is currently only an idea to get closer to the real solution
2008-05-11T11:32:18 <ThomasWaldmann> but let's first discuss about how it works at all
2008-05-11T11:32:25 <ThomasWaldmann> so why does that fix anything?
2008-05-11T11:33:50 <dreimark> it seems to me that there is something doing 1sec steps
2008-05-11T11:34:15 <dreimark> it can be from the filesystem too
2008-05-11T11:34:31 <dreimark> my ramdisk is of fstype ext2
2008-05-11T11:37:11 <ThomasWaldmann> did you try insert a 2s sleep into the test (between changing stuff and testing effect)?
2008-05-11T11:38:37 <dreimark> I did after every test 1s sleep and this doesn't change
2008-05-11T11:38:52 <dreimark> I have had 5-6 failing tests always
2008-05-11T11:39:05 <ThomasWaldmann> not after, but within
2008-05-11T11:39:29 <dreimark> right, after create, append ...
2008-05-11T11:55:10 <ThomasWaldmann> btw, create_random_string does not do what the function name tells it does
2008-05-11T11:56:39 <ThomasWaldmann> and why does it create 3000 random strings by default?
2008-05-11T11:59:19 <dreimark> may be it is a bit to excessive I do wanted to have a larger number
2008-05-11T11:59:55 <dreimark> if there are timing problems I don't wanted to test with to small list
2008-05-11T12:02:36 <ThomasWaldmann> creating slower timing by using a bigger list is broken
2008-05-11T12:03:07 <ThomasWaldmann> so please: change the name to something sane, and default count 10 or so
2008-05-11T12:04:34 <dreimark> create_name_list ?
2008-05-11T12:05:53 <ThomasWaldmann> create_random_string_list
2008-05-11T12:06:23 <dreimark> :)
2008-05-11T12:06:34 <ThomasWaldmann> and also fix param names and comments
2008-05-11T12:06:54 <ThomasWaldmann> it is just strings, not restricted to "names"
2008-05-11T12:07:34 <ThomasWaldmann> remove the # long list comment
2008-05-11T12:07:51 <dreimark> heh too much details now
2008-05-11T12:09:50 <ThomasWaldmann> the nuke_user code should be moved upwards to the stuff dealing with users
2008-05-11T12:14:31 <dreimark> how long can I make the list of strings in testAppendingGroupPage
2008-05-11T12:17:26 <ThomasWaldmann> what's the point of that test? testing scalability?
2008-05-11T12:17:33 <dreimark> right
2008-05-11T12:19:50 <ThomasWaldmann> ok, then add a comment telling this
2008-05-11T12:20:33 <ThomasWaldmann> and please, use more sane variable names
2008-05-11T12:20:45 <ThomasWaldmann> name = [u" * %s" % text for text in create_random_string(name_len=20, count=30000)]
2008-05-11T12:20:52 <dreimark> changed len to length already
2008-05-11T12:21:05 <ThomasWaldmann> this is not obfuscated python contest :P
2008-05-11T12:21:59 <dreimark> create_random_string_list(length=20, count=1) ?
2008-05-11T12:22:34 <ThomasWaldmann> maybe explain to yourself what your code is about and then choose speaking names
2008-05-11T12:23:20 <ThomasWaldmann> "name" is a bad name for page content
2008-05-11T12:23:37 <ThomasWaldmann> "text" is a bad name for a group member
2008-05-11T12:25:32 <ThomasWaldmann> and if the default length of 14(?) is no problem, you don't need to explicitely change it to 20
2008-05-11T12:29:39 <dreimark> i don't want to test for a repeated value
2008-05-11T12:32:11 <ThomasWaldmann> 52^14 is unlikely to have dupes
2008-05-11T12:36:50 <dreimark> unlikly but that one should be different by length too. (15) is long enough
2008-05-11T12:38:29 * dreimark will continue after lunch
2008-05-11T17:43:03 <dreimark> re
2008-05-11T17:49:22 <ThomasWaldmann> wb dreimark :)
2008-05-11T17:49:41 * dreimark tries to do all the wanted changes
2008-05-11T17:50:28 <dreimark> (sometimes I'm to generic with names)
2008-05-11T17:58:21 <ThomasWaldmann> hmm, we have wikiserverconfig.py, so we could also mv moin.py wikiserver.py
2008-05-11T18:03:49 <dreimark> that's a good idea easier to get/explain dependencies too
2008-05-11T18:04:25 * ThomasWaldmann renames it
2008-05-11T18:06:12 <CIA-53> Reimar Bauer <rb.proj AT googlemail DOT com> default * 3575:baa5bfe1b8e1 1.7/MoinMoin/_tests/ (__init__.py test_wikidicts.py): test_wikidicts: refactored tests to more useful names, changed defaults for the string list
2008-05-11T18:20:40 <dreimark> ThomasWaldmann: caching uses os.path.getmtime
2008-05-11T18:20:54 <dreimark> which does not work for eventlog
2008-05-11T18:23:52 <dreimark> http://hg.moinmo.in/moin/1.7/rev/d62fd466d810
2008-05-11T18:35:38 <CIA-53> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 3576:08a8965162fe 1.7/ (docs/CHANGES wikiserver.py wikiserverlogging.conf moin.py): rename ./moin.py to wikiserver.py, add wikiserverlogging.conf to configure its logging
2008-05-11T18:35:39 <CIA-53> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 3577:6b1b444b45f9 1.7/MoinMoin/_tests/ (__init__.py test_wikidicts.py): merged main
2008-05-11T18:37:42 <ThomasWaldmann> dreimark: caching is not related to eventlog
2008-05-11T18:38:27 <dreimark> I know but the time in eventlog was better as mtime of the file
2008-05-11T18:39:59 <ThomasWaldmann> btw, comments as # don't use event_log.date() are pretty useless if you don't tell why
2008-05-11T18:56:05 <CIA-53> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 3578:9cb46c423384 1.7/wikiserverlogging.conf: logging: change the stderr handler to log at DEBUG level (we control the logging with the logger's level)
2008-05-11T19:05:07 <ThomasWaldmann> hmm, moin script command misses logging configuration
2008-05-11T19:10:09 <ThomasWaldmann> hmm, maybe we should rather use PYTHONPATH and MOINLOGGING than editing script files
2008-05-11T19:15:00 * dreimark has a solution for the caching problem
2008-05-11T19:15:19 <dreimark> (of course for discussion)
2008-05-11T19:15:32 <dreimark> or not
2008-05-11T19:15:55 <dreimark> sh.t
2008-05-11T19:16:19 <dreimark> (should run tests several times)
2008-05-11T19:43:27 <dreimark> grzywacz: hi, do you have added your changes to the changes file?
2008-05-11T19:43:31 <dreimark> http://moinmo.in/MoinMoinTodo/Release%201.7.0
2008-05-11T19:57:31 <grzywacz> dreimark, jabber notification is mentioned there
2008-05-11T19:57:56 <grzywacz> However, the HelpOnNotification page seems to have disappeared.
2008-05-11T19:58:04 <grzywacz> Or am I targeting a wrong wiki instance?
2008-05-11T19:58:56 <dreimark> http://master17.moinmo.in/HelpOnNotification
2008-05-11T19:58:57 <grzywacz> Ah, master17.
2008-05-11T19:59:22 <grzywacz> dreimark, what other "xmpp work" do you need mentioned in changes?
2008-05-11T19:59:43 <grzywacz> The GHOP tasks were technical and not really visible to end users.
2008-05-11T20:00:07 <ThomasWaldmann> i meant ghop when writing that
2008-05-11T20:00:40 <ThomasWaldmann> but we also have a section for developers :)
2008-05-11T20:01:02 <grzywacz> GHOP was basically 2 tasks: determining installation requirements for the jabber bot running on windows and implementing entity capabilities, which was split into 3 tasks IIRC.
2008-05-11T20:02:29 <ThomasWaldmann> and what do we do with entity capabilities? :)
2008-05-11T20:03:18 <dreimark> "Interactive Jabber bot allows to per..." is not described
2008-05-11T20:03:24 <grzywacz> We store them to avoid resending service discovery requests to every client that gets online.
2008-05-11T20:04:18 <ThomasWaldmann> ok, so it was kind of a fix
2008-05-11T20:04:31 <grzywacz> More like an enhancement to make the bot more robust.
2008-05-11T20:04:32 <ThomasWaldmann> then just edit the todo page and remove those lines
2008-05-11T20:04:48 <ThomasWaldmann> and see also the notification help page todo
2008-05-11T20:04:59 <dreimark> the hints for "Jabber servers" is missing too
2008-05-11T20:06:10 <grzywacz> dreimark, sorry..?
2008-05-11T20:06:31 <grzywacz> dreimark, you mean traffic limits?
2008-05-11T20:07:26 <grzywacz> ThomasWaldmann, http://moinmo.in/RicardoFang/Issue84 do you want this copied to master17 or somehow merged with HelpOnNotification?
2008-05-11T20:09:48 <dreimark> grzywacz: http://hg.moinmo.in/moin/1.7-jabber-knowak/file/f5896c05c7ea/docs/CHANGES.jabber
2008-05-11T20:10:09 <dreimark> I was refering to the entries below New Features there
2008-05-11T20:11:43 <dreimark> ThomasWaldmann: we still have the detailed description mode of CHANGES and will compress it later to a less verbose one
2008-05-11T20:11:48 <dreimark> as we did for 1.6
2008-05-11T20:11:53 <ThomasWaldmann> grzywacz: well, at least it should not stay at that url, but moved to a global page
2008-05-11T20:12:14 <ThomasWaldmann> i guess moinmoin wiki is better than master as usually not many people update win32 stuff
2008-05-11T20:12:55 <dreimark> so everything related to jabber should go into the main CHANGES file
2008-05-11T20:12:58 <grzywacz> First of all I can't edit HelpOnNotification on master17 even though I seem to be in the Editors group.
2008-05-11T20:13:55 <grzywacz> dreimark, CHANGES.jabber ended up in HelpOnNotification
2008-05-11T20:14:01 <grzywacz> dreimark, it's not even merged in 1.7
2008-05-11T20:14:18 <dreimark> ah ok
2008-05-11T20:49:02 <dreimark> ThomasWaldmann: new idea to fix the dict caching trouble
2008-05-11T20:49:12 <dreimark> and it is only a proof of concept
2008-05-11T20:49:16 <dreimark> http://paste.pocoo.org/show/49439/
2008-05-11T20:49:33 <dreimark> btw. I have had trouble to upload this file to MM
2008-05-11T20:52:09 <dreimark> 0 dict failures on my (ram)disk
2008-05-11T20:57:59 * dreimark dinner
2008-05-11T20:58:01 <ThomasWaldmann> dreimark: if you read it anyway, checking for its mtime is pointless
2008-05-11T21:02:20 <gizmach> moin
2008-05-11T21:04:16 <ThomasWaldmann> hi melita
2008-05-11T21:06:49 <gizmach> hi thomas :)
2008-05-11T21:10:01 <dreimark> hi melita
2008-05-11T21:10:10 <dreimark> ThomasWaldmann: scan_end_time: 1210532961.911472. mtime: 1210532961.953721.
2008-05-11T21:11:34 <dreimark> the problem is mtime is slightly wrong or less precision digits of file mtime
2008-05-11T21:15:59 <gizmach> hi Reimar
2008-05-11T21:18:22 <ThomasWaldmann> dreimark: that's ok
2008-05-11T21:19:04 <dreimark> if i do return mtime after reading the whole file it is going wrong too
2008-05-11T21:19:17 <dreimark> (the old mtime)
2008-05-11T21:20:21 <dreimark> I mean it looks like we need the "scan_end_time" entry of the file
2008-05-11T21:21:01 <dreimark> gizmach: how was your birthday party? did you have nice weather too?
2008-05-11T21:21:02 <ThomasWaldmann> we usually do mtime checks to avoid reading a file if it has nothing new
2008-05-11T21:21:06 * dreimark is sunburned
2008-05-11T21:21:22 <ThomasWaldmann> so if you read the file, you don't need mtime any more...
2008-05-11T21:21:25 <gizmach> dreimark: it was great weather and we had a great time
2008-05-11T21:22:00 <gizmach> I'm still trying to face myself that tomorrow is monday and I need to start working
2008-05-11T21:22:10 <ThomasWaldmann> esselle on #moin has package pages problems
2008-05-11T21:22:31 <ThomasWaldmann> gizmach: no free day tomorrow?
2008-05-11T21:22:53 <ThomasWaldmann> or do you mean moin work? :)
2008-05-11T21:22:58 <dreimark> mtime is more often used as only for the wikidicts files
2008-05-11T21:23:07 <gizmach> ThomasWaldmann: no I have college and some laboratories, also I mean moin work :)
2008-05-11T21:23:21 <gizmach> ThomasWaldmann: do you have some holliday or something tomorrow?
2008-05-11T21:23:40 <ThomasWaldmann> dreimark: that makes it even worse if your read the content there
2008-05-11T21:24:14 <dreimark> it is not sane currently only proof of concept I told before
2008-05-11T21:27:59 <dreimark> do you know which precision file mtime has for the different platforms?
2008-05-11T21:28:03 <dreimark> ThomasWaldmann:
2008-05-11T21:30:50 <ThomasWaldmann> can be as worse as 2s on win32 iirc
2008-05-11T21:33:50 <dreimark> I think for dict pages we need then the stored "scan_end_time"
2008-05-11T21:34:42 <dreimark> Are there other critical page/ache files
2008-05-11T21:34:48 <dreimark> +c
2008-05-11T21:37:50 <ThomasWaldmann> i first want that a definite bug is recognized, before stuff is "fixed"
2008-05-11T21:38:16 <ThomasWaldmann> or rather the root cause of the bug
2008-05-11T21:55:27 <dreimark> I think it happens at the path of caching.update
2008-05-11T21:55:52 <dreimark> we do create a tmp file do write all the stuff we need into that
2008-05-11T21:55:58 <dreimark> and we close it
2008-05-11T21:56:12 <dreimark> then it is renamed to the right file
2008-05-11T21:56:37 <dreimark> at that time it has the wrong mtime already an older one as the one written into the file
2008-05-11T21:57:22 <dreimark> s/older/newer/
2008-05-11T21:57:26 <dreimark> (bigger)
2008-05-11T22:05:34 <ThomasWaldmann> so it looks fresher than it is
2008-05-11T22:09:20 <dreimark> yeah, currently i try if i can set mtime after renaming
2008-05-11T22:10:07 <dreimark> but it seems os.utime likes only integers
2008-05-11T22:38:07 <CIA-53> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 3579:ff4afe930cb2 1.7/ (MoinMoin/log.py docs/CHANGES wikiserverlogging.conf): logging configuration: you can use MOINLOGGINGCONF env var now to override from where your logging conf is read, fix wikiserverlogging.conf to not create empty moin.log files
2008-05-11T22:42:05 <CIA-53> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 3580:fc2f05f2b0bd 1.7/ (9 files in 2 dirs): server adaptor scripts: make sys.path / logging configuration / comments consistent
2008-05-11T22:46:40 <CIA-53> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 3581:357cea7b7780 1.7/wiki/server/ (wikiserver.py moin moin.py): remove old moin daemonizing script (replaced by moin server standalone --start/stop), renamed wiki/server/moin.py to wikiserver.py and updated its content from the toplevel clone of it
2008-05-11T22:49:07 <dreimark> os.stat_float_times(True) should switch to float but do I have to set it everywhere?
2008-05-11T22:50:41 <ThomasWaldmann> if the filesystem mtime granularity is rather big, it doesn't help to use floats
2008-05-11T22:59:24 <CIA-53> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 3582:27ba7445b480 1.7/MoinMoin/log.py: logging: fix typos
2008-05-11T23:16:32 <dreimark> ThomasWaldmann: true. ext2 has 1sec that's an explaination for the time.sleep(1) too
2008-05-11T23:17:30 <dreimark> but i think getting that info from the file itself is better
2008-05-11T23:20:42 <dreimark> good night
2008-05-11T23:37:34 <CIA-53> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 3583:347dff3b8197 1.7/ (contrib/mailimport/wikimailimport wiki/server/moin): provide moin script command for people not using setup.py, use it for mailimport script
2008-05-11T23:37:37 <ThomasWaldmann> gn dreimark
MoinMoin: MoinMoinChat/Logs/moin-dev/2008-05-11 (last edited 2008-05-10 22:15:02 by IrcLogImporter)