首页澳门新葡亰官方网站 › PHP火速按行读取CSV大文件的封装类分享

PHP火速按行读取CSV大文件的封装类分享

$csvreader = new CsvReader($csv_file);

具体实现方法:复制代码
代码如下:$csv_file = 'path/bigfile.csv';$spl_object = new
SplFileObject;$spl_object->seek;echo
$spl_object->key();2、如何快速获取CSV大文件的数据?

 代码如下

补充:有一些朋友使用的是excel文档,这样这个程序就不可以使用了,我们可以使用phpexcel插件来读取excel文件并写入到数据库中去哦。

 

print_r;

$csv_file = 'path/bigfile.csv';
$spl_object = new SplFileObject($csv_file, 'rb');
$spl_object->seek(filesize($csv_file));
echo $spl_object->key();

用SplFileObject来处理大数据的csv文件效率高

1、如何快速获取CSV大文件的总行数?

$csv_file = 'path/bigfile.csv';

 

个人觉得写的还行,高手莫见笑。

 private function _file_valid($file = '') {
  $file = $file ? $file : $this->csv_file;
  if(!$file || !file_exists($file)) {
   return false;
  }
  if(!is_readable($file)) {
   return false;
  }
  return true;
 }

$csvreader = new CsvReader;

class CsvReader {
 private $csv_file;
 private $spl_object = null;
 private $error;
 
 public function __construct($csv_file = '') {
  if($csv_file && file_exists($csv_file)) {
   $this->csv_file = $csv_file;
  }
 }
 
 public function set_csv_file($csv_file) {
  if(!$csv_file || !file_exists($csv_file)) {
   $this->error = 'File invalid';
   return false;
  }
  $this->csv_file = $csv_file;
  $this->spl_object = null;
 }
 
 public function get_csv_file() {
  return $this->csv_file;
 }
 
 private function _file_valid($file = '') {
  $file = $file ? $file : $this->csv_file;
  if(!$file || !file_exists($file)) {
   return false;
  }
  if(!is_readable($file)) {
   return false;
  }
  return true;
 }
 
 private function _open_file() {
  if(!$this->_file_valid()) {
   $this->error = 'File invalid';
   return false;
  }
  if($this->spl_object == null) {
   $this->spl_object = new SplFileObject($this->csv_file,
'rb');
  }
  return true;
 }

这是magento脚本的另外一种写法。

仍然使用PHP的SplFileObject类,通过seek方法实现快速定位。

public function get_data($length = 0, $start = 0) {if {return
false;}$length = $length ? $length : $this->get_lines();$start =
$start - 1;$start = ? 0 : $start;$data =
array();$this->spl_object->seek;while ($length-- &&
!$this->spl_object->eof {$data[] =
$this->spl_object->fgetcsv();$this->spl_object->next();}return
$data;}public function get_lines() {if {return
false;}$this->spl_object->seek(filesize;return
$this->spl_object->key();}public function get_error() {return
$this->error;}}调用方法如下:复制代码
代码如下:include('CsvReader.class.php');

至于数据如何入库本文不再详细讲述。

<?php
require_once 'shell/abstract.php';
class Faarao_Import_Customer extends Mage_Shell_Abstract
{
protected $_files = array();
protected $_datas = array();
public function __construct() {
parent::__construct();
set_time_limit(0);
if($this->getArg('file')) {
$this->_files = array_merge(
$this->_files,
array_map(
'trim',
explode(',',
$this->getArg('file'))
)
);
foreach ($this->_files as
$key=>$file) {
$extension = self::get_extension($file);
if($extension != 'csv'){
unset($this->_files[$key]);
}
}
}
if(empty($this->_files)){
die(self::usageHelp());
}
}
// Shell script point of entry
public function run() {
self::getDataFromCsv();
if(empty($this->_datas)){
die("not found data in csv ! \r\n");
}
$emailBooks = array();
foreach ($this->_datas as $name => $datas) {
echo "filename: {$name} =======================\r\n";
$importNum = 0;
foreach ($datas as $key => $data) {
if(empty(trim($data[4]))){
continue;
}
$customerData['firstname'] = trim($data[0]);
$customerData['lastname'] = trim($data[1]);
$customerData['phone'] = trim($data[2]);
$customerData['mobile'] = trim($data[3]);
$customerData['email'] = trim($data[4]);
$customerData['company'] = trim($data[5]);
$customerData['billing_address'] =
$data[6] . $data[7] . $data[8];
$customerData['billing_postcode'] =
sprintf("%05d",trim($data[9]));
$customerData['billing_city'] = trim($data[10]);
$customerData['billing_country'] = trim($data[11]);
$customerData['shipping_address'] = $data[12] . $data[13]
.$data[14];
if(empty($data[15]) && is_numeric($data[16])){
$customerData['shipping_postcode'] =
sprintf("%05d",trim($data[16]));
$customerData['shipping_city'] = trim($data[17]);
$customerData['shipping_country'] = trim($data[18]);
}else{
$customerData['shipping_澳门新葡亰官方网站,postcode'] = sprintf("%05d",
trim($data[15]));;
$customerData['shipping_city'] = trim($data[16]);
$customerData['shipping_country'] = trim($data[17]);
}
// $customerData['email'] = 'zouhongzhao@126.com';
$customerData['country'] = 'Finland';
$customerData['password'] =
self::randomkeys(10);
 
 
 
echo "customer email {$customerData['email']} ...\r\n";
print_r($customerData);
$customer = Mage::getModel('customer/customer');
$customer->setWebsiteId(Mage::app()->getWebsite()->getId());
$customer->loadByEmail($customerData['email']);
if(!$customer->getId()) {
echo "insert ... \r\n";
$customer->setEmail($customerData['email']);
$customer->setFirstname($customerData['firstname']);
$customer->setLastname($customerData['lastname']);
$customer->setPassword($customerData['password']);
}else{
echo "update ... \r\n";
}
try {
$customer->save();
$customer->setConfirmation(null);
$customer->save();
//Make a "login" of new customer
Mage::getSingleton('customer/session')->loginById($customer->getId());
$importNum ++;
$emailBooks[$customerData['email']] = array(
'firstname'=>$customerData['firstname'],
'lastname'=>$customerData['lastname'],
'customer_mage_id'=>$customer->getId(),
'passwd'=>$customerData['password']
);
 
echo "customer save ok !\r\n";
}
catch (Exception $ex) {
echo "customer save fail !\r\n";
continue;
}
 
if(trim($customerData['billing_address']) ==
trim($customerData['shipping_address'])
&& $customerData['billing_postcode'] ==
$customerData['shipping_postcode']
&& $customerData['billing_city'] ==
$customerData['shipping_city']){
$same_address = array (
'firstname' => $customerData['firstname'],
'lastname' => $customerData['lastname'],
'street' => $customerData['billing_address'],
'company' => $customerData['company'],
'city' => $customerData['billing_city'],
'region_id' => '',
'region' => '',
'postcode' => $customerData['billing_postcode'],
'country_id' => 'FI',
'telephone' => $customerData['phone'],
);
 
$customAddress = Mage::getModel('customer/address');
//$customAddress = new Mage_Customer_Model_Address();
$customAddress->setData($same_address)
->setCustomerId($customer->getId())
->setIsDefaultBilling('1')
->setIsDefaultShipping('1')
->setSaveInAddressBook('1');
try {
$customAddress->save();
echo "sameAddress save ok !\r\n";
}
catch (Exception $ex) {
echo "sameAddress save fail !\r\n";
continue;
}
}else{
$billing_address = array (
'firstname' => $customerData['firstname'],
'lastname' => $customerData['lastname'],
'street' => $customerData['billing_address'],
'company' => $customerData['company'],
'city' => $customerData['billing_city'],
'region_id' => '',
'region' => '',
'postcode' => $customerData['billing_postcode'],
'country_id' => 'FI',
'telephone' => $customerData['phone'],
 
);
self::setBillingAddress($billing_address,$customer);
$shipping_address = array (
'firstname' => $customerData['firstname'],
'lastname' => $customerData['lastname'],
'street' => $customerData['shipping_address'],
'company' => $customerData['company'],
'city' => $customerData['shipping_city'],
'region_id' => '',
'region' => '',
'postcode' => $customerData['shipping_postcode'],
'country_id' => 'FI',
'telephone' => $customerData['phone'],
);
self::setShippingAddress($shipping_address,$customer);
}
// die;
}
echo "import num: {$importNum} =======================\r\n";
}
//save passwd
$fp =
fopen('customer_record.log',
'w');
fwrite($fp, json_encode($emailBooks));
fclose($fp);
 
}
 
public function setBillingAddress($data,$customer){
$customerAddress = Mage::getModel('customer/address');
 
if ($defaultShippingId = $customer->getDefaultBilling()){
$customerAddress->load($defaultShippingId);
} else {
$customerAddress
->setCustomerId($customer->getId())
->setIsDefaultBilling('1')
->setSaveInAddressBook('1')
;
 
$customer->addAddress($customerAddress);
}
 
try {
$customerAddress
->addData($data)
->save()
;
echo "BillingAddress save ok !\r\n";
} catch(Exception $e){
// Mage::log('Address Save Error::' . $e->getMessage());
echo "BillingAddress save fail !\r\n";
}
}
 
public function setShippingAddress($data,$customer){
$customerAddress = Mage::getModel('customer/address');
 
if ($defaultShippingId = $customer->getDefaultShipping()){
$customerAddress->load($defaultShippingId);
} else {
$customerAddress
->setCustomerId($customer->getId())
->setIsDefaultShipping('1')
->setSaveInAddressBook('1')
;
 
$customer->addAddress($customerAddress);
}
 
try {
$customerAddress
->addData($data)
->save()
;
echo "ShippingAddress save ok !\r\n";
} catch(Exception $e){
// Mage::log('Address Save Error::' . $e->getMessage());
echo "ShippingAddress save fail !\r\n";
}
}
 
public function randomkeys($length) {
$returnStr='';
$pattern =
'1234567890abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLOMNOPQRSTUVWXYZ';
for($i = 0; $i < $length; $i ++) {
$returnStr .= $pattern {mt_rand ( 0, 61 )};
}
return $returnStr;
}
public function getDataFromCsv(){
$this->_files = array_unique($this->_files);
$csvCustomers = array();
foreach ($this->_files as $filename) {
echo "current file: {$filename}\r\n";
setlocale(LC_ALL, 'en_US.UTF-8');
$content = file_get_contents($filename);
$data =
mb_detect_encoding()($content,
'UTF-8', true);
// $data =
iconv("CP1257","UTF-8",
$content);
file_put_contents($filename, $content);
 
// fclose($handle);
// print_r($content);die;
$basename = basename($filename,".csv");
$data = array();
$tmp = array();
$spl_object = new SplFileObject($filename, 'rb');
$spl_object->seek(filesize($filename));
$start = 0;
$num = $spl_object->key();
$spl_object->seek($start);
while ($num-- && !$spl_object->eof()) {
$data[] = $spl_object->fgetcsv();
$spl_object->next();
}
foreach ($data as $key => $values) {
if($key == 0){
continue;
}
$mergeValue = explode(';',implode(';', $values));
// if(count($mergeValue) != 20){
// continue;
// }
array_push($tmp,$mergeValue);
}
 
$this->_datas[$basename] = $tmp;
}
return $this;
}
 
public function get_extension($filename){
return pathinfo($filename,PATHINFO_EXTENSION);
}
// Usage instructions
public function usageHelp()
{
return <<<USAGE
Usage: php tetuan_customer_import.php --file a.csv,b.csv
\n
USAGE;
}
}
// Instantiate
$shell = new Faarao_Import_Customer();
// Initiate script
$shell->run();

 public function get_csv_file() {
  return $this->csv_file;
 }

$data = $csvreader->get_data;

为了成功将CSV文件里的数据导入数据库,分批处理是非常必要的。

 

print_r($data);

1、如何快速获取CSV大文件的总行数?

调用方法如下:

 代码如下

复制代码 代码如下:

办法一:直接获取文件内容,使用换行符进行拆分得出总行数,这种办法对小文件可行,处理大文件时不可行;办法二:使用fgets一行一行遍历,得出总行数,这种办法比办法一好一些,但大文件仍有超时的可能;办法三:借助SplFileObject类,直接将指针定位到文件末尾,通过SplFileObject::key方法获取总行数,这种办法可行,且高效。

print_r($data);

复制代码 代码如下:

$line_number = $csvreader->get_lines();

 public function get_data($length = 0, $start = 0) {
  if(!$this->_open_file()) {
   return false;
  }
  $length = $length ? $length : $this->get_lines();
  $start = $start - 1;
  $start = ($start < 0) ? 0 : $start;
  $data = array();
  $this->spl_object->seek($start);
  while ($length-- && !$this->spl_object->eof()) {
   $data[] = $this->spl_object->fgetcsv();
   $this->spl_object->next();
  }
  return $data;
 }
 
 public function get_lines() {
  if(!$this->_open_file()) {
   return false;
  }
  $this->spl_object->seek(filesize($this->csv_file));
  return $this->spl_object->key();
 }
 
 public function get_error() {
  return $this->error;
 }
}

$csv_file = 'path/bigfile.csv';
$start = 100000;  // 从第100000行开始读取
$num = 100;    // 读取100行
$data = array();
$spl_object = new SplFileObject($csv_file, 'rb');
$spl_object->seek($start);
while ($num-- && !$spl_object->eof()) {
 $data[] = $spl_object->fgetcsv();
 $spl_object->next();
}
print_r($data);

仍然使用PHP的SplFileObject类,通过seek方法实现快速定位。复制代码 代码如下:$csv_file =
'path/bigfile.csv';$start = 100000;// 从第100000行开始读取$num = 100;//
读取100行$data = array();$spl_object = new
SplFileObject;$spl_object->seek;while ($num-- &&
!$spl_object->eof {$data[] = $spl_object->fgetcsv;}print_r;

上述函数对500M以内的文件进行过测试,运行通畅,对于1GB的文件发现有点慢了,于是再接着找方法。

复制代码 代码如下:

echo $line_number, chr;

具体实现方法:

转载本站文章请注明出处:澳门新葡亰官方网站 http://www.radioritmo-bl.com/?p=1125

上一篇:

下一篇:

相关文章