日志管理(6) 写一个超级简单的日志查询页面

把所有的业务日志通过scribe收集上来后,需要给项目开发人员一个查询业务日志的页面,方便他们查日志也给我们运维节约时间,我们可以用节约的时间做更多更有意义的事。写的这个页面很简单后端用的是Django,前端展示用的是基于bootstrap的Metronic模板,因为不需要权限控制也不要登录验证故没有用到数据库。
用一个业务日志查询页面来介绍下,其它业务日志查询的逻辑都一样甚至渲染的模板都几乎一样,唯一不同的是渲染模板传递的参数有小差异。
实现的功能很简单:可以查询每天最新的日志如最新100行;根据日期查询某天日志最后100行;根据一个或多个关键字查询日志并显示最新的200行结果;每次查询的结果都可以通过页面的下载按钮下载到本地供日后分析;页面上显示的结果按时间排序,最新的显示在最前面。
1、渲染模板,每个业务日志模板都由一个基础模板和各自独有的模板构成。
基础模板bash.html

<!DOCTYPE html>
<!--[if IE 8]> <html lang="en" class="ie8"> <![endif]-->
<!--[if IE 9]> <html lang="en" class="ie9"> <![endif]-->
<!--[if !IE]><!--> <html lang="en"> <!--<![endif]-->
<!-- BEGIN HEAD -->
<head>
	<meta charset="utf-8" />
    <title>{{privatetitle}}</title>
	<meta content="width=device-width, initial-scale=1.0" name="viewport" />
	<meta content="" name="description" />
	<meta content="" name="author" />
	<!-- BEGIN GLOBAL MANDATORY STYLES -->
	<link href="{{STATIC_URL}}/css/bootstrap.min.css" rel="stylesheet" type="text/css"/>
	<link href="{{STATIC_URL}}/css/bootstrap-responsive.min.css" rel="stylesheet" type="text/css"/>
	<link href="{{STATIC_URL}}/css/font-awesome.min.css" rel="stylesheet" type="text/css"/>
	<link href="{{STATIC_URL}}/css/style-metro.css" rel="stylesheet" type="text/css"/>
	<link href="{{STATIC_URL}}/css/style.css" rel="stylesheet" type="text/css"/>
	<link href="{{STATIC_URL}}/css/style-responsive.css" rel="stylesheet" type="text/css"/>
	<link href="{{STATIC_URL}}/css/default.css" rel="stylesheet" type="text/css" id="style_color"/>
	<link href="{{STATIC_URL}}/css/uniform.default.css" rel="stylesheet" type="text/css"/>
	<!-- END GLOBAL MANDATORY STYLES -->
    <!-- BEGIN PAGE LEVEL STYLES -->
    <link rel="stylesheet" type="text/css" href="{{STATIC_URL}}/css/datepicker.css" />
    <link href="{{STATIC_URL}}/css/glyphicons.css" rel="stylesheet" />
	<!-- END PAGE LEVEL STYLES -->
	<link rel="shortcut icon" href="{{STATIC_URL}}/image/favicon.ico" />
</head>
<!-- END HEAD -->
<!-- BEGIN BODY -->
<body class="page-header-fixed">
	<!-- BEGIN HEADER -->
	<div class="header navbar navbar-inverse navbar-fixed-top">
		<!-- BEGIN TOP NAVIGATION BAR -->
		<div class="navbar-inner">
			<div class="container-fluid">
				<!-- BEGIN LOGO -->
				<a class="brand" href="/seek/">
				<img src="{{STATIC_URL}}/image/logo.png" alt="logo" />
				</a>
				<!-- END LOGO -->
				<!-- BEGIN RESPONSIVE MENU TOGGLER -->
				<a href="javascript:;" class="btn-navbar collapsed" data-toggle="collapse" data-target=".nav-collapse">
				<img src="{{STATIC_URL}}/image/menu-toggler.png" alt="" />
				</a>          
				<!-- END RESPONSIVE MENU TOGGLER -->
				<!-- BEGIN TOP NAVIGATION MENU -->
				<ul class="nav pull-right">
					<!-- BEGIN USER LOGIN DROPDOWN -->
					<li class="dropdown user">
						<a href="#" class="dropdown-toggle" data-toggle="dropdown">
						<img alt="" src="{{STATIC_URL}}/image/linux.jpg" />
						<span class="username">SomeBody</span>
						<i class="icon-angle-down"></i>
						</a>
						<ul class="dropdown-menu">
							<li><a href="#"><i class="icon-tasks"></i> My Home</a></li>
							<li class="divider"></li>
							<li><a href="#"><i class="icon-key"></i> Log Out</a></li>
						</ul>
					</li>
					<!-- END USER LOGIN DROPDOWN -->
				</ul>
				<!-- END TOP NAVIGATION MENU -->
			</div>
		</div>
		<!-- END TOP NAVIGATION BAR -->
	</div>
	<!-- END HEADER -->
	<!-- BEGIN CONTAINER -->
	<div class="page-container row-fluid">
		<!-- BEGIN SIDEBAR -->
		<div class="page-sidebar nav-collapse collapse">
			<!-- BEGIN SIDEBAR MENU -->        
			<ul class="page-sidebar-menu">
				<li>
					<!-- BEGIN SIDEBAR TOGGLER BUTTON -->
					<div class="sidebar-toggler hidden-phone"></div>
					<!-- BEGIN SIDEBAR TOGGLER BUTTON -->
				</li>
				<li class="active ">
                <a href="javascript:;"><i class="icon-table"></i><span class="title">日志查询</span><span class="selected"></span><span class="arrow open"></span>
                </a>
					<ul class="sub-menu">
                        <li class={{seek01}}>
                        <a href="/seek/aaaaa/">A日志</a>
						</li>
                        <li class={{seek02}}>
                        <a href="/seek/bbbbb/nginx-php/">B日志</a>
						</li>
                        <li class={{seek03}}>
                        <a href="/seek/ccccc/">C日志</a>
						</li>
                        <li class={{seek04}}>
                        <a href="/seek/ddddd/services/">D日志</a>
						</li>
                        <li class={{seek05}}>
                        <a href="/seek/eeeee/nginx-php/">E日志</a>
						</li>
					</ul>
				</li>
			</ul>
			<!-- END SIDEBAR MENU -->
		</div>
		<!-- END SIDEBAR -->
		<!-- BEGIN PAGE -->
		<div class="page-content">
			<!-- BEGIN SAMPLE PORTLET CONFIGURATION MODAL FORM-->
			<div id="portlet-config" class="modal hide">
				<div class="modal-header">
					<button data-dismiss="modal" class="close" type="button"></button>
					<h3>portlet Settings</h3>
				</div>
				<div class="modal-body">
					<p>Here will be a configuration form</p>
				</div>
			</div>
			<!-- END SAMPLE PORTLET CONFIGURATION MODAL FORM-->
			<!-- BEGIN PAGE CONTAINER-->
			<div class="container-fluid">
				<!-- BEGIN PAGE HEADER-->
				<div class="row-fluid">
                    <div class="span12">
                        {% block rightareadown %}<br><h3>请选择项目</h3>{% endblock %}
					</div>
				</div>
				<!-- END PAGE CONTENT-->
			</div>
			<!-- END PAGE CONTAINER-->
		</div>
		<!-- END PAGE --> 
	</div>
	<!-- END CONTAINER -->
	<!-- BEGIN FOOTER -->
	<div class="footer">
		<div class="footer-inner">
			Copyright (c) 2014 日志查询平台. All rights reserved
		</div>
		<div class="footer-tools">
			<span class="go-top">
			<i class="icon-angle-up"></i>
			</span>
		</div>
	</div>
	<!-- END FOOTER -->
	<!-- BEGIN JAVASCRIPTS(Load javascripts at bottom, this will reduce page load time) -->
	<!-- BEGIN CORE PLUGINS -->
	<script src="{{STATIC_URL}}/js/jquery-1.10.1.min.js" type="text/javascript"></script>
	<script src="{{STATIC_URL}}/js/jquery-migrate-1.2.1.min.js" type="text/javascript"></script>
	<!-- IMPORTANT! Load jquery-ui-1.10.1.custom.min.js before bootstrap.min.js to fix bootstrap tooltip conflict with jquery ui tooltip -->
	<script src="{{STATIC_URL}}/js/jquery-ui-1.10.1.custom.min.js" type="text/javascript"></script>      
	<script src="{{STATIC_URL}}/js/bootstrap.min.js" type="text/javascript"></script>
	<!--[if lt IE 9]>
	<script src="{{STATIC_URL}}/js/excanvas.min.js"></script>
	<script src="{{STATIC_URL}}/js/respond.min.js"></script>  
	<![endif]-->   
	<script src="{{STATIC_URL}}/js/jquery.slimscroll.min.js" type="text/javascript"></script>
	<script src="{{STATIC_URL}}/js/jquery.blockui.min.js" type="text/javascript"></script>  
	<script src="{{STATIC_URL}}/js/jquery.cookie.min.js" type="text/javascript"></script>
	<script src="{{STATIC_URL}}/js/jquery.uniform.min.js" type="text/javascript" ></script>
	<!-- END CORE PLUGINS -->
    <!-- BEGIN PAGE LEVEL PLUGINS -->
    <script type="text/javascript" src="{{STATIC_URL}}/js/bootstrap-datepicker.js"></script>
    <script type="text/javascript" src="{{STATIC_URL}}/js/date.js"></script>
    <!-- END PAGE LEVEL PLUGINS -->
	<!-- BEGIN PAGE LEVEL SCRIPTS -->
	<script src="{{STATIC_URL}}/js/app.js"></script>
    <script src="{{STATIC_URL}}/js/form-components.js"></script>
	<script>
		jQuery(document).ready(function() {
		// initiate layout and plugins
        App.init();
        FormComponents.init();
		});
	</script>
	<!-- END PAGE LEVEL SCRIPTS -->
<script type="text/javascript">  var _gaq = _gaq || [];  _gaq.push(['_setAccount', 'UA-37564768-1']);  _gaq.push(['_setDomainName', 'keenthemes.com']);  _gaq.push(['_setAllowLinker', true]);  _gaq.push(['_trackPageview']);  (function() {    var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;    ga.src = ('https:' == document.location.protocol ? 'https://' : 'http://') + 'stats.g.doubleclick.net/dc.js';    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);  })();</script></body>
<!-- END BODY -->
</html>

每个项目还有自己独有的模板且继承bash.html,独立模板用来重写{% block rightareadown %}{% endblock %}块,独立的模板几乎一样就举一例。

{% extends "base.html" %}
{% block rightareadown %}
<ul class="breadcrumb">
    <li>
        <i class="icon-home"></i>
        <a href="/seek/">日志查询</a> 
	    <i class="icon-angle-right"></i>
	</li>
	<li><a href="/seek/aaaaa/">A日志</a></li>
</ul>
<div class="row-fluid">
    <div class="span12">
        <!-- BEGIN PORTLET-->
        <div class="portlet box grey">
            <div class="portlet-title">
                <div class="caption">
                    <i class="icon-reorder"></i>填入必要的查询条件
                </div>
                <div class="tools">
                    <a href="javascript:;" class="collapse"></a>
                    <a href="javascript:;" class="remove"></a>
                </div>
            </div>
        <div class="portlet-body form">
        <!-- BEGIN FORM-->
        <form action="/seek/aaaaa/" class="form-horizontal" enctype="multipart/form-data" method="post">
            <div class="control-group">
                <label class="control-label">选择查询日期(默认当天)</label>
                <div class="controls">
                    <div class="input-append date date-picker" data-date-format="mm/dd/yyyy" data-date-viewmode="years">
                        <input class="m-wrap m-ctrl-medium date-picker" readonly size="16" type="text" value="" name="date"/><span class="add-on"><i class="icon-calendar"></i></span>
                    </div>
                </div>
            </div>
            <div class="control-group">
                <label class="control-label">选择要查询的日志</label>
                <div class="controls">
                    <select class="span6 m-wrap" multiple="multiple" data-placeholder="Choose a Category" tabindex="1" name="logdir" >
                        {% for dir in dirs %}
                        <option>{{ dir }}</option>
                        {% endfor %}
                    </select>
                </div>
            </div>
            <div class="control-group">
                <label class="control-label">Keys</label>
                <div class="controls">
                    <input type="text" class="span6 m-wrap" name="keys" placeholder="输入关键字,多个关键字用分号分隔(不输入关键字则获取当天最新的100行记录)">
                </div>
            </div>
            <div class="form-actions">
                <button type="submit" class="btn blue">Submit</button>
            </div>
            {% csrf_token %}
        </form>
        <!-- END FORM-->  
        </div>
    </div>
    <!-- END PORTLET-->
</div>

<div class="row-fluid">
    <div class="portlet box grey">
        <div class="portlet-title">
            <div class="caption">
                <i class="icon-reorder"></i>查询结果
            </div>
            <div class="tools">
                <a href="javascript:;" class="collapse"></a>
            </div>
        </div>
    <div class="portlet-body">
        <div class="control-group">
            <a href="{{durl}}" class="glyphicons no-js download_alt"><i></i>下载</a>
        </div>
    <div class="scroller" data-height="550px" data-always-visible="1">
        {{ value }}
        {% for mes in message %}
        {{ mes }}<br><br>
        {% endfor %}
    </div>
    </div>
    </div>
</div>
{% endblock %}

2、视图函数,接收前端页面的POST请求并根据不同的请求条件,使用日志查询函数searchKey, noKeys, readLine查询日志,然后将结果渲染后返回给前端页面。

#coding=utf-8
from django.shortcuts import render_to_response
from django.template import RequestContext
from os import listdir, path
from search import searchKey, noKeys, readLine
import time

def seekindex(request):
    avg = {'privatetitle': '日志查询', 'STATIC_URL': '/static'}
    return render_to_response('base.html', avg)

def seekvenus(request):
    message = ''
    durl = '#'
    value = ''
    logPath = r"/var/scribe_data/aaaaa"
    logList = listdir(logPath)
    dirs = []
    for dir in logList:
        dirs.append(dir.replace("1_", ''))
    dirs.sort()
    if request.method == 'POST':
        try:
            date = request.POST['date']
        except:
            pass
        try:
            logdir = request.POST['logdir']
        except:
            logdir = 'nodir'
        try:
            keys = request.POST['keys']
        except:
            pass
        if not date:
            date = time.strftime('%m/%d/%Y',time.localtime(time.time())) 
        if date and logdir and keys:
            #fname
            temp = date.split('/')
            dd = temp[1] 
            mm = temp[0] 
            yy = temp[2]
            fname = '/var/scribe_data/aaaaa/'+'1_'+logdir+'/'+'1_'+logdir+'-'+yy+'-'+mm+'-'+dd+'_00000'
            #keys
            keys = keys.split(';')
            #seek
            if not path.exists(fname):
                value = "当天的日志文件不存在"
            else:
                dfile = searchKey(fname, keys)
                durl = r'/static/tmp/' + dfile.split('/')[-1]
                message = readLine(dfile, False)
        elif logdir == 'nodir':
            value = '请选择日志'
        else:
            temp = date.split('/')
            dd = temp[1] 
            mm = temp[0] 
            yy = temp[2]
            fname = '/var/scribe_data/aaaaa/'+'1_'+logdir+'/'+'1_'+logdir+'-'+yy+'-'+mm+'-'+dd+'_00000'
            if not path.exists(fname):
                value = "当天的日志文件不存在"
            else:
                dfile = noKeys(fname) 
                durl = r'/static/tmp/' + dfile.split('/')[-1]
                message = readLine(dfile, True)
    avg = {'privatetitle': 'aaaaa日志查询', 'STATIC_URL': '/static', 'message':message, 'seek01':"active" ,'dirs':dirs, 'durl':durl, 'value':value}
    return render_to_response('aaaaa/seek.html', avg, context_instance=RequestContext(request))

3、日志查询函数。每次先将一定条件的查询结果保存到一个新的文件,然后再从查询结果中读取最新的100行显示,这个新的文件可以通过页面的下载链接上下载到本地。

#!/usr/bin/python
from re import search, sub
import logging
import commands 

def searchKey(fname, keys, user='Unknown'):
    logging.basicConfig(filename = '/var/www/logseek/log/seek.log', level = logging.INFO, filemode = 'a', format = '%(asctime)s - %(levelname)s: %(message)s')
    tmpdir = r'/var/www/logseek/static/tmp/' 
    resultFile = tmpdir + fname.split(r'/')[-1]+'-'+keys[0]+'-'+keys[-1]+'.log'
    file = open(fname)
    refile = open(resultFile, 'w')
    while True:
        lines = file.readlines(500000)
        if not lines:
            break
        for line in lines:
            for key in keys:
                row = search(key, str(line))
                if not row:
                    row = False
                    break
            if row:
                refile.write(line)
    file.close()
    refile.close()
    keyinfo = ''
    for key in keys:
        keyinfo = keyinfo + ':' + key
    logging.info(user + ':' + fname + keyinfo)
    return resultFile

def noKeys(fname, user='Unknown'):
    logging.basicConfig(filename = '/var/www/logseek/log/seek.log', level = logging.INFO, filemode = 'a', format = '%(asctime)s - %(levelname)s: %(message)s')
    tmpdir = r'/var/www/logseek/static/tmp/'
    resultFile = tmpdir + fname.split(r'/')[-1]+'-'+'new100.log'
    comand = 'tail -n 100 '+fname+' > '+ resultFile 
    try:
        stat, proStr = commands.getstatusoutput(comand)
    except:
        pass
    logging.info(user + ':' + fname + ' tail 100 lines')
    return resultFile

def readLine(fname, flag):
    if flag:
        comand = 'tail -n 100 '+fname
    else:
        comand = 'tail -n 200 '+fname
    try:
        stat, proStr = commands.getstatusoutput(comand)
    except:
        pass
    relist = proStr.split('\n')
    relist.reverse()
    return relist

4、最后看到的查询日志页面
logs-terrace
一个非常简单实用的日志查询页面,如果以后还有更多需求再慢慢堆代码。
文章出处:http://www.xiaomastack.com/2014/11/16/logs-terrace/

1 Comment

 Add your comment
  1. 呵呵,版主做的不错,期望能提供下代码包。

发表评论:

你的电子邮件地址将不会被公开.

89 − = 83