belachelijk langdurige updates

[webjunkie]

edit: opgelost in de nieuwste versie

ik heb de retrieve.php gescheduled zodat hij om de 30 minuten update.
dat werkt goed alleen duurt dat sommige keren belachelijk lang.. wel zo'n 5 tot 8 minuten

hij haalt dan heel erg veel comments binnen.
zoals hieronder: 39 spots en 187431 comments.

Code: Selecteer alles

C:\xampp\htdocs\spotweb>C:\xampp\php\php.exe -f C:\xampp\htdocs\spotweb\retrieve.php 
Retrieving new Spots from server...
Spots in database:	45198
Looking for articlenumber for messageid
Appr. Message count: 	408994
First message number:	2133088
Last message number:	2542082
Current message:	2542044

Retrieving 2542083 till 2543083 (parsed 28 headers, retrieved 28 full spots, verified 28, of 39 spots)
Processed a total of 39 spots
Finished retrieving spots.

Retrieving new comments from server...
Looking for articlenumber for messageid
Appr. Message count: 	229104
First message number:	147968
Last message number:	377072
Current message:	147967

Retrieving 147974 till 148974, found 2 comments
Retrieving 148974 till 149974, found 0 comments
Retrieving 149919 till 150919, found 6 comments
Retrieving 150360 till 151360, found 2 comments
Retrieving 151360 till 152360, found 0 comments
Retrieving 152170 till 153170, found 4 comments
Retrieving 153170 till 154170, found 0 comments
Retrieving 154170 till 155170, found 0 comments
Retrieving 155170 till 156170, found 0 comments
Retrieving 155984 till 156984, found 4 comments
Retrieving 156806 till 157806, found 8 comments
Retrieving 157774 till 158774, found 10 comments
Retrieving 158738 till 159738, found 14 comments
Retrieving 159689 till 160689, found 10 comments
Retrieving 160383 till 161383, found 8 comments
Retrieving 161128 till 162128, found 4 comments
Retrieving 161699 till 162699, found 2 comments
Retrieving 162699 till 163699, found 0 comments
Retrieving 163651 till 164651, found 4 comments
Retrieving 164639 till 165639, found 28 comments
Retrieving 165639 till 166639, found 359 comments
Retrieving 166640 till 167640, found 964 comments
Retrieving 167641 till 168641, found 1001 comments
Retrieving 168642 till 169642, found 1001 comments
Retrieving 169643 till 170643, found 1001 comments
Retrieving 170644 till 171644, found 1001 comments
Retrieving 171645 till 172645, found 1001 comments
Retrieving 172646 till 173646, found 1001 comments
Retrieving 173647 till 174647, found 1001 comments
Retrieving 174648 till 175648, found 1001 comments
Retrieving 175649 till 176649, found 548 comments
Retrieving 176650 till 177650, found 308 comments
Retrieving 177651 till 178651, found 1001 comments
Retrieving 178652 till 179652, found 1001 comments
Retrieving 179653 till 180653, found 1001 comments
Retrieving 180654 till 181654, found 1001 comments
Retrieving 181655 till 182655, found 1001 comments
Retrieving 182656 till 183656, found 1001 comments
Retrieving 183657 till 184657, found 1001 comments
Retrieving 184658 till 185658, found 1001 comments
Retrieving 185659 till 186659, found 1001 comments
Retrieving 186660 till 187660, found 1001 comments
Retrieving 187661 till 188661, found 1001 comments
Retrieving 188662 till 189662, found 1001 comments
Retrieving 189663 till 190663, found 1001 comments
Retrieving 190596 till 191596, found 933 comments
Retrieving 191596 till 192596, found 0 comments
Retrieving 192596 till 193596, found 0 comments
Retrieving 193596 till 194596, found 0 comments
Retrieving 194596 till 195596, found 0 comments
Retrieving 195597 till 196597, found 700 comments
Retrieving 196598 till 197598, found 1001 comments
Retrieving 197599 till 198599, found 1001 comments
Retrieving 198600 till 199600, found 1001 comments
Retrieving 199601 till 200601, found 1001 comments
Retrieving 200602 till 201602, found 1001 comments
Retrieving 201603 till 202603, found 1001 comments
Retrieving 202604 till 203604, found 993 comments
Retrieving 203605 till 204605, found 1001 comments
Retrieving 204606 till 205606, found 1001 comments
Retrieving 205607 till 206607, found 1001 comments
Retrieving 206608 till 207608, found 1001 comments
Retrieving 207609 till 208609, found 1001 comments
Retrieving 208610 till 209610, found 1001 comments
Retrieving 209611 till 210611, found 1001 comments
Retrieving 210612 till 211612, found 1001 comments
Retrieving 211613 till 212613, found 1001 comments
Retrieving 212614 till 213614, found 1001 comments
Retrieving 213615 till 214615, found 1001 comments
Retrieving 214616 till 215616, found 1001 comments
Retrieving 215617 till 216617, found 1001 comments
Retrieving 216618 till 217618, found 1001 comments
Retrieving 217619 till 218619, found 1001 comments
Retrieving 218620 till 219620, found 1001 comments
Retrieving 219621 till 220621, found 1001 comments
Retrieving 220622 till 221622, found 1001 comments
Retrieving 221623 till 222623, found 1001 comments
Retrieving 222624 till 223624, found 1001 comments
Retrieving 223625 till 224625, found 1001 comments
Retrieving 224626 till 225626, found 1001 comments
Retrieving 225627 till 226627, found 1001 comments
Retrieving 226628 till 227628, found 1001 comments
Retrieving 227629 till 228629, found 1001 comments
Retrieving 228630 till 229630, found 1001 comments
Retrieving 229631 till 230631, found 1001 comments
Retrieving 230632 till 231632, found 1001 comments
Retrieving 231633 till 232633, found 1001 comments
Retrieving 232534 till 233534, found 225 comments
Retrieving 233535 till 234535, found 315 comments
Retrieving 234536 till 235536, found 1001 comments
Retrieving 235537 till 236537, found 1001 comments
Retrieving 236538 till 237538, found 1001 comments
Retrieving 237539 till 238539, found 1001 comments
Retrieving 238540 till 239540, found 999 comments
Retrieving 239541 till 240541, found 1001 comments
Retrieving 240542 till 241542, found 1000 comments
Retrieving 241543 till 242543, found 1001 comments
Retrieving 242544 till 243544, found 1001 comments
Retrieving 243545 till 244545, found 1001 comments
Retrieving 244546 till 245546, found 1001 comments
Retrieving 245547 till 246547, found 1001 comments
Retrieving 246548 till 247548, found 730 comments
Retrieving 247549 till 248549, found 229 comments
Retrieving 248548 till 249548, found 385 comments
Retrieving 249549 till 250549, found 664 comments
Retrieving 250550 till 251550, found 899 comments
Retrieving 251551 till 252551, found 834 comments
Retrieving 252541 till 253541, found 342 comments
Retrieving 253463 till 254463, found 176 comments
Retrieving 254464 till 255464, found 913 comments
Retrieving 255465 till 256465, found 1001 comments
Retrieving 256466 till 257466, found 1001 comments
Retrieving 257467 till 258467, found 1001 comments
Retrieving 258468 till 259468, found 1001 comments
Retrieving 259469 till 260469, found 1001 comments
Retrieving 260470 till 261470, found 1001 comments
Retrieving 261471 till 262471, found 1001 comments
Retrieving 262472 till 263472, found 692 comments
Retrieving 263452 till 264452, found 311 comments
Retrieving 264411 till 265411, found 112 comments
Retrieving 265402 till 266402, found 135 comments
Retrieving 266370 till 267370, found 89 comments
Retrieving 267371 till 268371, found 203 comments
Retrieving 268369 till 269369, found 163 comments
Retrieving 269369 till 270369, found 90 comments
Retrieving 270370 till 271370, found 120 comments
Retrieving 271332 till 272332, found 161 comments
Retrieving 272333 till 273333, found 383 comments
Retrieving 273334 till 274334, found 1001 comments
Retrieving 274335 till 275335, found 629 comments
Retrieving 275303 till 276303, found 122 comments
Retrieving 276304 till 277304, found 307 comments
Retrieving 277305 till 278305, found 1001 comments
Retrieving 278306 till 279306, found 973 comments
Retrieving 279307 till 280307, found 681 comments
Retrieving 280308 till 281308, found 999 comments
Retrieving 281309 till 282309, found 1001 comments
Retrieving 282310 till 283310, found 1001 comments
Retrieving 283311 till 284311, found 998 comments
Retrieving 284309 till 285309, found 993 comments
Retrieving 285310 till 286310, found 978 comments
Retrieving 286311 till 287311, found 1001 comments
Retrieving 287312 till 288312, found 1001 comments
Retrieving 288313 till 289313, found 1001 comments
Retrieving 289314 till 290314, found 1001 comments
Retrieving 290315 till 291315, found 1001 comments
Retrieving 291316 till 292316, found 1001 comments
Retrieving 292317 till 293317, found 1001 comments
Retrieving 293318 till 294318, found 1001 comments
Retrieving 294319 till 295319, found 1001 comments
Retrieving 295320 till 296320, found 1001 comments
Retrieving 296321 till 297321, found 1001 comments
Retrieving 297322 till 298322, found 1001 comments
Retrieving 298323 till 299323, found 1001 comments
Retrieving 299324 till 300324, found 1001 comments
Retrieving 300325 till 301325, found 1001 comments
Retrieving 301326 till 302326, found 1001 comments
Retrieving 302327 till 303327, found 1001 comments
Retrieving 303328 till 304328, found 1001 comments
Retrieving 304329 till 305329, found 1001 comments
Retrieving 305330 till 306330, found 1001 comments
Retrieving 305493 till 306493, found 35 comments
Retrieving 306493 till 307493, found 0 comments
Retrieving 307493 till 308493, found 0 comments
Retrieving 308493 till 309493, found 0 comments
Retrieving 309494 till 310494, found 836 comments
Retrieving 310495 till 311495, found 1001 comments
Retrieving 311496 till 312496, found 1001 comments
Retrieving 312497 till 313497, found 1001 comments
Retrieving 313498 till 314498, found 1001 comments
Retrieving 314499 till 315499, found 1001 comments
Retrieving 315500 till 316500, found 1001 comments
Retrieving 316501 till 317501, found 1001 comments
Retrieving 317502 till 318502, found 1001 comments
Retrieving 318503 till 319503, found 1001 comments
Retrieving 319504 till 320504, found 1001 comments
Retrieving 320505 till 321505, found 1001 comments
Retrieving 321506 till 322506, found 1001 comments
Retrieving 322507 till 323507, found 1001 comments
Retrieving 323508 till 324508, found 1001 comments
Retrieving 324509 till 325509, found 984 comments
Retrieving 325510 till 326510, found 1001 comments
Retrieving 326511 till 327511, found 1001 comments
Retrieving 327512 till 328512, found 1001 comments
Retrieving 328513 till 329513, found 1001 comments
Retrieving 329514 till 330514, found 1001 comments
Retrieving 330515 till 331515, found 1001 comments
Retrieving 331516 till 332516, found 1001 comments
Retrieving 332517 till 333517, found 1001 comments
Retrieving 333518 till 334518, found 1001 comments
Retrieving 334519 till 335519, found 1001 comments
Retrieving 335520 till 336520, found 1001 comments
Retrieving 336521 till 337521, found 1001 comments
Retrieving 337522 till 338522, found 1001 comments
Retrieving 338523 till 339523, found 1001 comments
Retrieving 339524 till 340524, found 1001 comments
Retrieving 340525 till 341525, found 1001 comments
Retrieving 341526 till 342526, found 1001 comments
Retrieving 342527 till 343527, found 1001 comments
Retrieving 343528 till 344528, found 1001 comments
Retrieving 344529 till 345529, found 1001 comments
Retrieving 345530 till 346530, found 1001 comments
Retrieving 346531 till 347531, found 1001 comments
Retrieving 347532 till 348532, found 1001 comments
Retrieving 348533 till 349533, found 1001 comments
Retrieving 349534 till 350534, found 1001 comments
Retrieving 350535 till 351535, found 1001 comments
Retrieving 351536 till 352536, found 1001 comments
Retrieving 352537 till 353537, found 1001 comments
Retrieving 353538 till 354538, found 1001 comments
Retrieving 354539 till 355539, found 1001 comments
Retrieving 355540 till 356540, found 1001 comments
Retrieving 356541 till 357541, found 1001 comments
Retrieving 357542 till 358542, found 1001 comments
Retrieving 358543 till 359543, found 1001 comments
Retrieving 359544 till 360544, found 1001 comments
Retrieving 360545 till 361545, found 1001 comments
Retrieving 361546 till 362546, found 1001 comments
Retrieving 362547 till 363547, found 1001 comments
Retrieving 363548 till 364548, found 1001 comments
Retrieving 364549 till 365549, found 1001 comments
Retrieving 365550 till 366550, found 1001 comments
Retrieving 366551 till 367551, found 1001 comments
Retrieving 367552 till 368552, found 1001 comments
Retrieving 368553 till 369553, found 1001 comments
Retrieving 369554 till 370554, found 1001 comments
Retrieving 370555 till 371555, found 1001 comments
Retrieving 371556 till 372556, found 1001 comments
Retrieving 372557 till 373557, found 1001 comments
Retrieving 373558 till 374558, found 1001 comments
Retrieving 374559 till 375559, found 1001 comments
Retrieving 375560 till 376560, found 1001 comments
Retrieving 376561 till 377561, found 1001 comments
Retrieving 377213 till 378213, found 652 comments
Processed a total of 187431 comments
Finished retrieving comments.

daarna update hij weer 1 of 2 keer normaal en en haalt hij maar een paar honderd comments binnen en geen paar honderd duizend.

W7 + xampp + mysql

[Blauwbek]

ja bij mij duurt dat ook aardig lang, doe je er een truncate vooraf of niet?
overgens haalt ie natuurlijk ook comments voor andere spots binnen he, niet alleen die 39

[webjunkie]

nee, ik doe geen truncate.
dat is eigenlijk m'n follow-up vraag.. hoe kan je de commentsxover tabel wat uitdunnen?
er staan nu nog maar 45k spots in de database maar de commentsxover tabel is nu al 560 MB.

overgens haalt ie natuurlijk ook comments voor andere spots binnen he

maar 187 duizend in 30 minuten lijkt me toch wel vrij veel

[Blauwbek]

ik heb het zo (en het duurt bij mij <5 minuten): <- overgens heb ik kpn met +/- 2 weken retentie
ik truncate eerst:
leeg.php:

Code: Selecteer alles

<?php
require_once "settings.php";
$host = $settings['db']['host'];
$db = $settings['db']['dbname'];
$user = $settings['db']['user'];
$pass = $settings['db']['pass'];

$con = mysql_connect("$host","$user","$pass");

mysql_select_db("$db", $con);

if (!$con)
  {
  die('Could not connect: ' . mysql_error());
  }
  
if (mysql_query("TRUNCATE TABLE spots",$con))
  {
  echo "Empty\n";
  }
else
  {
  echo "Error: " . mysql_error();
  }
  
  mysql_close($con);
?>

(wat hij bij mij automatisch doet omdat ik eerst dit laat doen):
retrieve.php:

Code: Selecteer alles

require_once "leeg.php";

[webjunkie]

als ik het goed begrijp verwijder je zo al je spots.
de spots en nzb's wil ik eigenlijk wel houden alleen de comments mogen wel wat minder.

het zou mooi zijn als er zo'n 10 comments per spot van de spots ouder dan 1 maand overblijven.

edit:

dit is trouwens de log van 30 minuten later en dan haalt hij maar 949 comments binnen.
over een half uur weer 190.000 ?

Code: Selecteer alles

C:\xampp\htdocs\spotweb>C:\xampp\php\php.exe -f C:\xampp\htdocs\spotweb\retrieve.php 
Retrieving new Spots from server...
Spots in database:	45226
Looking for articlenumber for messageid
Appr. Message count: 	409017
First message number:	2133088
Last message number:	2542105
Current message:	2542055

Retrieving 2542106 till 2543106 (parsed 23 headers, retrieved 23 full spots, verified 23, of 51 spots)
Processed a total of 51 spots
Finished retrieving spots.

Retrieving new comments from server...
Looking for articlenumber for messageid
Appr. Message count: 	229541
First message number:	147968
Last message number:	377509
Current message:	376561

Retrieving 377510 till 378510, found 949 comments
Processed a total of 949 comments
Finished retrieving comments.

[webjunkie]

ik lees net dat deze bug gefixt moet zijn in de laatste versie.

[Blauwbek]

het duurt hier nu ook belachelijk lang, net geupdate (lekker begin zo)...